Public Stats API

Too many posts to reply to, so not going to bother quoting them all.

Not sure if they were directed towards me personally or not, but it feels like it since I've been in the discussion.

I'm not hating on Sota Assist. I respect that people put hard work into coding a useful tool and even the fact that they are storing data retrieved for redundancy. That's great for people who want lots of information all the time.

Me personally, I'm only interested in finding an easy way to say lookup where a specific recipe or crafting material or blueprint dropped... so that I know where I need to go to try to get one for myself. The reason I wanted to get a list of the archetypes or exact item names is that I want to search for a single specific item (like the Rotunda spider room blueprint) instead of looking for a type of item (ie. just searching for "blueprint"). I don't mind the delay of the free site or the fact that it would take me several attempts to get the information I need. I'm in no rush and if I only get a handful of results instead of a database of hundreds of results... that's just fine.

I don't care about PVP stats or player kills or any of that kind of stuff. Just some simple searches on where a few specific things were/are found in the game.

So, in my case, I'm just saying I don't feel like it's worth it to pay for a service/tool that provides more than I need, especially when all I need is possible for me from a free to use website... or ... to get myself once I figure out how to do it.

 

That is not a complaint. He is saying a small number of people are hammering ElasticSearch and if these people do not throttle their usage Portalium may need to shut down our external access to it.

What they have provided is not optimized in any way for massive query/user loads. Setting up a proper API (likely to include some sort of caching to keep costs down) is an incredibly heavy and ongoing software engineering task. The groups that are making money from this (though i doubt this is very much money) should be particularly conscientious since they are using a free resource.

This said, there is most definitely value in caching the data (all-be-it at the cost of significantly more complexity). Any set of applications that naively polls the ElasticSearch instance on behalf of thousands of potential users will perform far worse (remember the 5 second limit is IP based). Back to the first statement though, setting up more resilient and scale-able APIs is HARD engineering. We know who these people are, so let's just have a conversation about what Portalium can handle. I have a couple decades of experience with these types of issues, so let me know how i can help.

Edit to add: The data amounts we are talking about here are actually quite small compared to other modern systems. There are brute force ways to scale Lucene if a group was able to take on the associated cost for clusters/shards. Ultimately though, modeling the data and its access is usually (not always!) more cost efficient for longer lasting initiatives.

 

I don't think anyone was saying the complaint wasn't JUSTIFIED. Clearly, you shouldn't be calling the API every time a user loads your page. Thus far it seems that @Undone just pointing that out has been enough (or I assume it has, as he hasn't brought it up again). When it came up originally (if memory serves) there was discussion about making us register for a key to access the API. If it becomes a problem again, that remains an option.

Why are we even discussing this?

 

Is there a way to map the SceneName to a more recognizable place name? For example, where is "POT_forest_metropolis_01_template?" @Undone

 

POT names are hidden on purpose.

 

@Undone The public API is nice! But have you checked out the websocket plugin option for elasticsearch? I really recommend playing around with websocket and or SocketIO.
Subscribing to a websocket as a client is much more friendly for us and also at the end for your API server. The load and performance will be much better.

 

@Undone
How about an API to search player vendors (opt-in only and no prices, of course)?

 

Who wouldn't "opt-in" to list their vendor ? I mean, you got him there to sell, right ?
No prices ?
Hmmm, I would prefer them - but I can understand how people who buy/resell wouldn't be a fan.
Too bad, IMO.........

 

Are fishing events being tracked in the system? Is it just falling under loot? It would be nice if it was under its own LocationEvent type, so it's easy to segment out.

Thanks!

 

Does the API still work?
I am getting 503 Service Unavailable.
* Connected to shroudoftheavatar.com (54.155.190.206) port 9200 (#0)
No server is available to handle this request.

 

It could be up today, but Chris mentioned no later than next week on Markee's stream. I'm sure life might get in the way, but it sounds like the task is in the queue to get done.

 

@Titania Xylia

Fishing tab isn't done yet, but to give you an idea: