yefim · 11 years ago

The GitHub repo is inaccessible because it is private.

zuck9 · 11 years ago

What's the Arc rendering engine? Google didn't help

minimaxir · 11 years ago

How does this differ from the Algolia HN API in terms of data access? (https://hn.algolia.com/api) I was able to download all HN data recently with ease using that endpoint. Authentication?

EDIT: After looking at the documentation there are two new aspects of the Firebase API not in the Algolia API:

1) Ability to see deleted/dead stories.

2) Endpoint for user data.

Question to kogir/dang: Has the "delay" field (Delay in minutes between a comment's creation and its visibility to other users) always been there?

bambax · 11 years ago

Any kind of rate limitations we should be made aware of?

jxm262 · 11 years ago

Yay! I've been wanting something like this to come out. I've been playing around with some new tech stacks and built a css replacer of hacker news, but always wanted an actual api to make it easier.

http://jmaat.me/hn

There's a bunch of css pages that come out for hacker news, but I couldn't find anything that aggregates them. This will be alot easier to extend and customize the site.

I'm not seeing any api's for the jobs or show sections though? Hopefully this might come in the future?

josephwegner · 11 years ago

I really appreciate giving a 3 week heads up before moving to a new frontend structure. It's a nice gesture, but I have this horrible feeling that there's only about a 10% chance that my Hacker News app gets updated in time.

I know you can't not iterate because people are scraping, but it does stink. At least this will make everything more future-proof going forward.

However, it may be nice to give a bit more heads up than 3 weeks. I know a lot of apps can take ~2 weeks to get through the review process for iOS.

burntcookie90 · 11 years ago

Very cool, I'm glad I prepped my android app for this :D

One question: why the choice of returning everything as an ID, on mobile, this will require a lot of very small network requests.

ChuckMcM · 11 years ago

Oh this will be cool, and I really look forward to being able to read HN on a phone or tablet in the future without all the zooming and scrolling!

It will be interesting to see if it has an impact on site traffic, how much of that traffic is scrapers today?

angersock · 11 years ago

Why do you have all the times in Unix time instead of ISO-8601 UTC?

EDIT: Also, the "about" value in the users api appears to be truncated. Compare...

https://hacker-news.firebaseio.com/v0/user/angersock#

...with...

https://news.ycombinator.com/user?id=angersock

EDIT2: Note that the JSON is correct, but the preview in the firebase API seems to be broken.

EDIT3: No issue tracker on the Github? Laaaame.

kasperset · 11 years ago

This is a very good news. Made my day. Thank you.

jamest · 11 years ago

[Firebase founder here] This is pretty exciting for us, we're glad kogir, dang, kevin and sctb chose to expose HN's data through Firebase. We're seen quite a few startups (and big companies like Nest) do this, since building, maintaining, and documenting a public API often isn't a easy task.

dpcan · 11 years ago

As soon as someone uses this API to create a replica of the site where everything else is the same, but the design is responsive, I want to know about it :)

christiangenco · 11 years ago

Oh man you guys, patio11 has generated massive amounts of content: https://hacker-news.firebaseio.com/v0/user/patio11.json?prin...

I count 8,483 submissions. I'm sure there's something interesting to be done with all of this data. A word frequency chart?

---

Edit: So apparently there's a ruby gem that lets you feed it a body of text and generates pseudo-random phrases based on that text.

I present to you the patio11 impersonator: https://gist.github.com/christiangenco/e8d085e47479be0131e1

One of my favorites:

    A nice set of challenges -- kitty at a school with tens of thousands of bucks a year or less immediately.

Also, a word count on patio11's submissions: 1,052,351. For comparison, all 7 Harry Potter books total 1,084,170 words. patio11 has written the entire Harry Potter series worth of content on HN. Just... wow.

pbnjay · 11 years ago

Great news! Can we add endpoints for "Show HN" and "Ask HN" too though? I don't see an easy way to get them.

airlocksoftware · 11 years ago

This... is cool, but also kinda sucks for me. I've invested dozens of hours into writing an extremely complicated scraper for my Android version of HN.

https://play.google.com/store/apps/details?id=com.airlocksof...

The newest version (still under development, probably a month or two from release) adds support for displaying polls, linking to subthreads, and full write support (voting, commenting, submitting, etc). I'm fine with switching to a new API (Square's Retrofit will make it super easy to switch), but without submitting, commenting, and upvote support I have to disable a bunch of features I worked really hard on. Also it would've been cool to know this was coming about 3 months ago so I didn't waste my time.

Anyways, quick question on how it works -- when I query for the list of top stories

https://hacker-news.firebaseio.com/v0/topstories.json?print=...

it just returns a list of ids. Do I have to make a separate request for each story

https://hacker-news.firebaseio.com/v0/item/8863.json?print=p...)

to assemble them into a list for the front page, or am I missing something?

tlo · 11 years ago

For my twitter bot https://twitter.com/hn_bot_top1 I use http://api.ihackernews.com/ at the moment. This works but the site (and the API) is quite often not available. So I'll probably switch to the new API as soon as I have some spare time...

dabit3 · 11 years ago

I love firebase!

dimillian · 11 years ago

So, does it mean I can get the top stories, and then get a top story item with all the comment expanded ? I mean, at first it look like it just send me the id and I need to fetch the detail for each of them. Again, this is just looking at the REST API, not the iOS SDK for example.

I'll need to "convert" SwiftHN (https://github.com/Dimillian/SwiftHN) either to this new API or adapt my scrapping engine to the new site layout.

BenSS · 11 years ago

Great news, I've avoided touching my iPad reader because of the whole scraping issue to get at some of the data. Now I can justify updating for iOS8 + the new API. (just hope it gets approved in time)

Achshar · 11 years ago

I always thought HN was more of a one man job. I didn't realize there is an entire team working on it (full time?). Honestly there doesn't seem to be much to work on here. It's a simple site that rarely changes (at least on the front end).

tonysuper · 11 years ago

HN is getting a mobile-optimized site?

Best news I've heard all day!

Livven · 11 years ago

I've been working on a Hacker News client for Windows Phone over the past several weeks and am very close to an initial release, so I feel somewhat ambivalent about this.

On the one hand, of course it's great that HN is finally getting a proper API and also modernizing its markup (which is a mess even if you ignore all the tables – for example, the first paragraph in a comment usually isn't wrapped in <p> tags), but on the other hand this current v0 version is very lacking and impractical for a regular client application.

Since the top stories (limited to 100) and child comments are only available as a list of IDs a client app would have to make a separate HTTP request for every single item, which is obviously not something you'd want to do especially in a mobile environment. Other lists apart from the top stories (new, show, ask, best, active etc.) don't seem to be available at all right now.

Of course this is just the first version, and the documentation promises improvements over time – which I don't doubt at all – but there's no clear indication that the API will be at feature-parity with the current website, even excluding anything that requires authentication, by October 28. So this means that I – and other developers of client apps or unofficial APIs – will probably have to write new scraping code once the new rendering engine (which I assume refers to the website) arrives instead of being able to switch to the new API immediately.

Now I guess I might just be needlessly worried, especially since the blog post explicitly says that the new API "should hopefully making switching your apps fairly painless", but then why not wait until it's actually ready for that before making the announcement? Putting a half-baked API out there a few days/weeks (?) in advance before it's fully fleshed out doesn't seem all that helpful, at least to me.

projectileboy · 11 years ago

Listening to programmers complaining about the markup used to implement HN is a perfect example of tech folks not focusing on the users.

ossama · 11 years ago

Here's a simple example that displays the top story and votes using the Firebase JS SDK (and updates in realtime): http://jsfiddle.net/firebase/cm8ne9nh/

eevilspock · 11 years ago

If up/down vote data were included in the API, much needed experimentation on collaborative filtering would be made possible! This is Hacker News after all.

Right now one team, Ycombinator, is trying to fix important issues in the ranking and moderation of posts and comments. Many of us are frustrated by the increasing domination of popularity (and hatred) over quality and relevance. A lot of good submissions and comments are simply buried, never to be found. There is too much muck to have to wade through. The timing of posts and comments plays a much larger role than quality. I could go on and on.

Imagine a Netflix Prize-like flowering of experiments and collaboration, leveraging the hacker community's collective smarts and enthusiasm. Many of us have ideas, but right now are unable to test them. What a shame if a great idea dies on a notepad.

There are two possible issues with opening up voting data: gaming and privacy. If having vote data allows someone to game the front page, then only include it with some delay (2 days?) so that it could't be used to game the front page. This will still allow experimentation with collaborative filtering algorithms and the like.

My take on the privacy issue is that anonymity isn’t that important for a site like Hacker News:

1. Startup culture is about straight talk, putting your money where your mouth is, and open critical feedback, both in the giving and receiving. There are precedents for exposing voting data (e.g. Quora, Facebook, Stack Exchange).

2. HN is not aimed at political discussions or other topics where anonymity can be paramount.

3. Pseudonymity is sufficient for those who don’t want their votes and comments tied back to their actual identity.

Thoughts?

I would love to hear from others who yearn to experiment with alternate algorithms and strategies for improving Hacker News.

BigChiefSmokem · 11 years ago

Does anyone have a good solution on how to get around making a separate request for each Item? Is there somewhere we can pass an array of Item Ids? Is this planned for the future?

Thanks HN!

EDIT: I just read the post about using the Firebase SDK to do this efficiently.

dang · 11 years ago

To everyone asking about logged-in access and write access: this is just a first release! Where it goes from here will depend, in good iterative fashion, on what people want.

kevin · 11 years ago

Oh and if you guys build anything cool with the API, please let us know at api@ycombinator.com.

me_bx · 11 years ago

I hope they'll add an article search (by keyword) feature soon.

comeonnow · 11 years ago

I built a scraper around 3 years ago (been through a few usernames since then), and I've had to change it once 3 months ago because the HTML output added quotes around HTML attributes.

Even though it's read only, I'll continue to use my scraper rather than the API simple because it's one request, rather than the API would require one request for the top IDs and then one call per story, so it would be 31 calls instead of just 1.

Unless I'm missing something, it seems fairly poorly designed for top stories, and non existent for new stories.

------

EDIT: Looks like I missed the text about updating to a new rendering system in 3 weeks time, and to iterate designs faster to allow mobile friendly theming. Looks Like I WILL be updating to use the API

markrickert · 11 years ago

Nice! I'll be changing my Mac system tray app HackerBar over to use this instead of some Objective-C scraping magic that it uses now: http://hackerbarapp.com/

jcampbell1 · 11 years ago

This makes it really easy to add average karma to the comment section for every user. For instance, you can paste the below into the console, and should add average karma data for each user.

    Array.prototype.forEach.call(document.querySelectorAll('a[href^=user]'),
    	function(v,k) { 
    		var s = document.createElement("script");
    		s.src = '//hacker-news.firebaseio.com/v0/user/' + v.innerHTML + '.json?callback=ud_' + k;   
    		document.head.appendChild(s);
    		window['ud_' + k] = function(user_data){
    			var avg_karma = user_data.karma / user_data.submitted.length;
    			v.innerHTML += ' (' + avg_karma.toFixed(1) + ')';
    		}
    	} 
    );

bambax · 11 years ago

Can we query new stories?

kogir · 11 years ago

Just pushed the ability to subscribe to all changes and new items: https://github.com/HackerNews/API/blob/master/README.md#live...

IgorPartola · 11 years ago

I am excited about this because now I can finally build a way to query my own posts and comments. I often times come across products/services/cool hacks on here that I vaguely remember, but cannot always locate. Using built-in search is kludgy, and I'd like to be able to do something more complex. Thanks for doing this!

davexunit · 11 years ago

Glad to see that HN will still be written in a lisp after these major upgrades.

lnanek2 · 11 years ago

Disappointing they are still going to use Arc even with the update to modern HTML. I despise how with Arc if I leave the page open during the day and try to click a link later the link has expired.

They should really update to a modern web framework at the same time. Big modern frameworks like Rails are making ridiculously awesome improvements like replacing page loads with XHR (quicker loading since JS/styles/etc is all loaded already, no screen flash, etc.) in a progressive enhancements manner.

So Arc can't even generate fully functional links, let alone keep up with modern web advancements.

awwstn · 11 years ago

This is so awesome – albeit a bit overdue ;)

I'm going to start diving into the API to build a simple, powerful "Google Alerts for HN" app on Assembly, and I'd love help from anyone who's interested: https://assembly.com/hn-monitor

There are some products like this out there, but they had to rely on scrapers and the HNSearch API, so I've always found them to be spotty. I think we can make something better.

oftenwrong · 11 years ago

I have been working on a toy project that uses the Algolia API. Maybe I'll switch it to this now.

EGreg · 11 years ago

This is excellent news, and allows me to make an app for YCombinator using our social app platform!

This should really energize the HN community.

danieldisu · 11 years ago

Can't wait to see all the apps that will be built with this, now that you don't have to scrape the horrible HN HTML!

hardwaresofton · 11 years ago

"Let there be apps".

Anyone with a Firefox phone that wants to work together on a HN client for it? FFOS has been lacking a good HN app.

calebm · 11 years ago

Why could you not have released it 1 day sooner? :) I just wrote a scraper: https://github.com/calebmadrigal/hn-tracker

calebm · 11 years ago

This story via the API: https://hacker-news.firebaseio.com/v0/item/8422599.json?prin...

calebm · 11 years ago

How can I find the current rank of a story via the API?

andrethegiant · 11 years ago

Any plan on having another field on the items route that can be queried besides id, e.g. url?

kolev · 11 years ago

At last! A hacker-friendly Hacker News! Yay!

jaredsohn · 11 years ago

Here is something I built with the Algolia API awhile back and just haven't gotten around to cleaning it up to post here.

It lets you download all comments/stories for a user as a JSON or CSV file, breaks down karma between comments and stories, and plots comment/story counts, karma, etc. over time on a line chart (clicking will show you the details via an hnsearch).

Also I built some npm modules so you can get this information via the commandline.

http://hnuser.herokuapp.com/.

Example: http://hnuser.herokuapp.com/user/tptacek/

The Chrome extension hasn't been updated for awhile (it just superimposes a small amount of this information on the user page).

bennyg · 11 years ago

I've been waiting forever for an API from HN, but unfortunately I will not be using it for my app (https://github.com/bennyguitar/News-YC---iPhone).

I've built a library for iOS (https://github.com/bennyguitar/libHN) that handles scraping, commenting, submitting, voting, etc pretty well and allows me to make as few web calls as necessary to use HN. It looks like I'd have to drop functionality and completely change the networking scheme to match this API - something I'm not willing to do yet.

Correct me if I'm wrong here, but to get every comment on a post, I'd have to recursively get each item for each child. Instead, right now, I can make one network request and get all comments for a story. Granted, I have to parse the HTML (which I hate), but it's a much cleaner solution than going through every item, checking the children and then getting those items ad infinitum. Again, I just glanced over the documentation, but that seems untenable to me.

dstaley · 11 years ago

Decided to recreate the Hacker News homepage using Ember and the new API. I was really pleased with how easy it was! https://realtimehackernews.firebaseapp.com/

wvl · 11 years ago

Would it be possible to cache the number of comments a story has? Or am I wrong in my understanding that the only way to find the number of comments a story has is to walk the tree of child items and maintain my own count?

ryanseys · 11 years ago

Smart move using Firebase. This instantly gives developers traction with the API and as a big API client guy, I love making clients but also very happy we'll have all the tools we need to get started using the API right off the bat. Considering Firebase doesn't have rate-limiting too, this means that the things people can build with this API are limitless.

wslh · 11 years ago

I would love a personal API too where I can download my own data (i.e: saved articles).

bch · 11 years ago

Does this encoding: &#x27;

have a name ? I can't actually find anything useful on the 'net.

stevekemp · 11 years ago

Until now I've been using the RSS feed to write a Maildir archive of stories.

It will be nice to write that directly via the API :)

jgill · 11 years ago

A very welcome addition.

deutronium · 11 years ago

I'm just wondering if you'd consider releasing a regular data dump, possibly via bittorrent.

As I'd like to do some analysis, but don't really want to thrash the API, downloading all data.

yawniek · 11 years ago

is there a simple way to stream all the updates or at least the new submissions/deletes by just subscribing to a Websocket?

shill · 11 years ago

I wrote a stupid simple wrapper and pushed it to PyPI. My excuse is that I needed to learn how to use setuptools today.

    pip install hackernews-python

    >>> from hackernews import HackerNews
    >>> hn = HackerNews()
    >>> hn.top_stories()
    [8422599, 8422087, 8422928, 8422581, 8423825...
    
    >>> hn.user('pg')
    {'delay': 2, 'id': 'pg', 'submitted': [7494555, 7494520, 749411...

    >>> hn.item(7494555)['title'])
    Hacker News API

    >>> hn.max_item()
    8424314

    >>> hn.updates()
    {'items': [8423690, 8424315, 8424299...], 'profiles': ['exampleuser',...]}

thekingshorses · 11 years ago

I added upvoting stories and comments feature to my ios/android app few weeks ago. And thinking about adding commenting.

http://hn.premii.com/about

Now I will have to remove that. Please keep current version of the YC site on a different subdomain.

quesera · 11 years ago

Great news! Will require a fair bit of retooling, but, ya know -- omelets, eggs, etc!

I see the topstories query to replace scraping /news: https://hacker-news.firebaseio.com/v0/topstories.json?print=...

And there must be a query to get the newest items instead of the current /newest.

Are there also new equivalents for /active, /best, /classic, /show, and /shownew?

I'll be happy to replace dodgy XPATH parsing with a proper API. Hope we won't lose these other views though!

elwell · 11 years ago

> the reason we released an API is so that we can start modernizing the markup on Hacker News

This is bigger news. No more tables!

fosk · 11 years ago

If you guys are already using Mashape, it has been released as a community API at https://www.mashape.com/community/hacker-news

na85 · 11 years ago

Please please please no javascript in the next HN iteration.

xtrumanx · 11 years ago

Perhaps keep the old site still running on a different URL so scrapers who can't get their act together in 3 weeks can just change the URL they're scraping from.

oldboyFX · 11 years ago

Ummm I have a question.

Why is so much preparation necessary to redesign like 3 simple templates. Shouldn't it take like ~10 hours for one person to do this? I'm talking about the front end.

Why wasn't hacker news optimized for mobile a long time ago?

:|

astevens · 11 years ago

I whipped up a quick test in Ruby

https://gist.github.com/astevens/627756eafbc6ca07e3dd

induscreep · 11 years ago

Finally, an AlienBlue caliber app for HN is now possible. Soon.

hrjet · 11 years ago

On a related tangent, since you mention modernizing the markup, can we please have markdown support in comments?

paulsutter · 11 years ago

Please do allow other sites to use HN logins. Then the community could develop useful sister services.

For example, a site where HN members can upvote and rate different development tools, libraries, IDEs, management tools, etc. All with backlinks to HN discussions. It's a great community and there are many ways we could share knowledge and experience.

jtwebman · 11 years ago

Nice time to makes some awesome data charts like most common subjects, etc...

stevenspasbo · 11 years ago

I spent about 20 minutes throwing this together, so it's VERY rough, but maybe it will turn into something useful. I'm primarily a java programmer but I've been wanting to teach myself more ruby, so here it goes.

https://github.com/stevenspasbo/Hackernews

EDIT: I should say, I started on this to create word clouds, but if anyone has any ideas, contribute!

sinwave · 11 years ago

It would be fun to whoop up an LDA analysis of this data.

ssorallen · 11 years ago

The Firebase JavaScript library makes make this impressively straightforward to use. I built a clone using React.js and Firebase's library. Because v0 of the API requires a request for each news story, it's not possible to use Firebase's React mixin yet.

https://github.com/ssorallen/hackernews-react

andreash · 11 years ago

Give me a HN trend chart for the words "pivot" and "mvp" please:)

izietto · 11 years ago

Yes!!! Now I can make my HN app! I don't like the ones that are on the market now (on Android platform)

x201s · 11 years ago

Glad to see there is an API. What about the ability to delete accounts? Why do I have to beg to delete an account?

aetherspawn · 11 years ago

I managed to take a picture when this got 1337 upvotes.

https://hostr.co/file/cHCoBCgwsjEc/2.PNG

r0nin · 11 years ago

I started a small package for golang here: https://github.com/cryptix/gohn

s9w · 11 years ago

I welcome the idea, but this barely qualifies as an API. The most useful part is the "current top stories" - but what timeframe exactly? Seems to be over 3 days at least and can't be customized. And even my test parsing of the 100 top stories took a good minute.

And that returns only the ids, nothing else. To get basic information like the score, title or url you have to lookup the ids individually. And even the story items do not contain such basic information as the number of comments. And you can't calculate it yourself since only the top comments are even returned (as ids of course). So you'll have to recursively dig through the comments to get the number.

This is even more curious as there is a very solid Algolia API where you can filter for submission time, story score, number of comments and even return a greater number of results + access page numbers to get even more.

To get the information of a single algolia api call you will need hundreds or thousands (in case of nested comments) "official" API calls. Hoping for updates

bajsejohannes · 11 years ago

Awesome!

On the top of my wishlist: Look up HN story by URL.

(From the comments, I learned about a different API that will suffice for now: https://hn.algolia.com/api/v1/search?query=google.com&restri... )

jingwen · 11 years ago

Wrote a simple ruby wrapper for this.

https://github.com/infinitus/hnrb

so_ · 11 years ago

I think quality of the v0 API is not so good for the following reasons.

1. Major functions are dropped (best, new, job, ask,...)

2. Useless response schema of the top stories API (should learn from Netflix's internal/public API design)

3. A short transition period

It's very hard to provide same responses to apps.

andrewstuart2 · 11 years ago

I'm definitely excited about the API and the future possibilities with it. Looks like a great start. I do have a few questions and suggestions, though.

Is there any chance of getting more than just the top 100 stories returned? I think it will be a lot more useful for api consumers if you can use a query parameter to set the limit (within reason, usually 1,000) and a number of results to skip. For now, scraping is still more desirable to me since I can retrieve any number of results in their current order.

Better yet, but more complex: a number to skip and a certain timestamp so I don't see the same article on two pages due to natural upvoting, downvoting, or rank decay.

Also, if there's any flexibility still with property names, I'd suggest these changes for clearer semantics: "deleted" -> "hidden" (since they're obviously not deleted) "by" -> "author" (for more clarity) "kids" -> "children" (the common convention)

Lekoaf · 11 years ago

Chiming in here and presenting my try at a Hacker News website using Angular JS and Bootstrap. https://github.com/lekoaf/HackerNews

My only problem with the API is that you need to do an awful lot of Ajax calls just to get something out of it. The topstories endpoint just gives you an array of IDs and then you need to do one Ajax call for each ID to get the story.

Oh well, the site is a work in progress. Not done yet.

bonzoq · 11 years ago

If someone would be interested in contributing to an OSS project to build an iOS HN client, please have a look at https://github.com/bonzoq/hniosreader.

tsenkov · 11 years ago

Are there API methods for New, Show, Ask, Jobs and Comments page items?

Or should I be picking them up from "/v0/topstories"?

bibonix · 11 years ago

What about search? Would be great to have an ability to search by URL or title fragments

zx33 · 11 years ago

Is there a way we can get all the submissions for a month or a date range?

benhamner · 11 years ago

Suggestions for improving the API, to make it more valuable for data mining and analytics. This assumes more historical data is available.

1. Provide a way to bulk download the data (that's a click, instead of scraping the API)

2. Add a field for the maximum position a story reached on the front page

3. Add the numerical score for the comment (at least on comments that are N days old, which won't interfere with the reason to hide the scores on the main comments)

Some other changes that would be awesome (but are less realistic) include:

4. A historical event log of votes (even better would be relating those votes back to users, but I imagine that's not going to happen for privacy reasons. An intermediate possibility would be a vote log connected back to anonymized user ids, assuming the anonymous id -> real user id mapping is difficult)

5. A historical event log of display position changes for stories & comments

6. An event log of pageviews with as much metadata as possible to release without infringing on privacy

chrismccoy · 11 years ago

http://jsfiddle.net/chrismccoy/cuz7ugL8/

quick fiddle to dump the json