How many times have you mentioned “coffee” on Twitter in the last six months? If you were archiving your tweets, the answer would be a simple API call away. While the Twitter API provides search capabilities, it comes with some restrictions, especially if you want to build anything with your tweets. You never have to worry about rate limits or API changes if you store each tweet as it happens. This post will show you how to do that with limited code.
I’ll use two free services to retrieve and store my tweets:
Prepare Your Database
Orchestrate is a database-as-a-service, so there is very little set up needed. You don’t need to spin up your own servers or create search indices. Just sign up for a free account.
To get your database ready for your tweets, all you need to do is create a new application. This will give you an API Key–click and copy the key to your clipboard. You’ll use it in the next step.
Optionally, you can also create a collection. I called mine tweets.
Listen for New Tweets
Zapier has the concepts of triggers and actions to be able to connect two services together. The trigger, in this example, will be a new tweet. The action is writing the tweet to Orchestrate.
- Make a new Zap with Twitter as the Trigger app and Web Hook as the Action app. The Web Hook action allows you to make a call to any API, like Orchestrate
- Now set the Trigger, “My Tweet,” and the Action, “PUT.”
- You’ll need to set up your Twitter account in Zapier so it knows whose tweets to grab.
Continue to the screen where you’ll match up Twitter to the Web Hook. That’s where you’ll be making an API call to Orchestrate.
Store Tweets in Orchestrate
Zapier now knows how to listen for your newest tweets. To store them in Orchestrate we need to match up Twitter to the Web Hook.
- In the URL field, add
https://api.orchestrate.io/v0/tweets/(where “tweets” is the name of your collection.
- Then click the button to Insert a Twitter field as the key for a particular tweet. This needs to be unique, so I chose “ID Str,” Twitter’s own identifier for a tweet.
- Payload type should be json
- The data is where you get to go wild storing whatever you want from your tweet. Just include the field name you want to use on the left side and insert the Twitter field on the right. Since you’re searching your tweets, at the very least get the “Text” of the tweet.
- The only other field you’ll need to change is the Basic Auth option. This is where you’ll paste your Orchestrate API Key, followed by the pipe character: |
- In Basic Auth terms, your API Key is your username and the password is blank.
- Finally, test the Zap with a Twitter Trigger.
After testing with one of the sample results, go to your Orchestrate dashboard to see the sample tweet added to your collection. From within your app, select the “tweets” collection. Then run a GET without providing a key to see the first few items (or perhaps only one at this point) in your collection.
You can switch to the Search option in the dashboard and get full text search access to your tweets.
Find Old Tweets and New Mentions
Zapier only listens for new tweets, which means with your Zap activated you’ll be collecting everything in Orchestrate from now on. What about all those amazing things you’ve already tweeted? You need to request your archive from Twitter and import it into Orchestrate manually.
Another next step you might take is archiving mentions, something that is not available in the typical Twitter archive. To do this, create a new Zap (or clone the one you just made) that with the “Search Mention” action.
Once you have all this data stored in Orchestrate, you’ll want to build something to search it. Check out our getting started with search to see code examples in your favorite language.