Public Stats API

Discussion in 'Developer Depot' started by Undone, May 9, 2017.

  1. Daxxe Diggler

    Daxxe Diggler Bug Brigade - Bug Hunter

    Messages:
    2,647
    Likes Received:
    5,631
    Trophy Points:
    153
    Gender:
    Male
    Location:
    Virtue Oasis - Hidden Vale
    Too many posts to reply to, so not going to bother quoting them all.

    Not sure if they were directed towards me personally or not, but it feels like it since I've been in the discussion.

    I'm not hating on Sota Assist. I respect that people put hard work into coding a useful tool and even the fact that they are storing data retrieved for redundancy. That's great for people who want lots of information all the time.

    Me personally, I'm only interested in finding an easy way to say lookup where a specific recipe or crafting material or blueprint dropped... so that I know where I need to go to try to get one for myself. The reason I wanted to get a list of the archetypes or exact item names is that I want to search for a single specific item (like the Rotunda spider room blueprint) instead of looking for a type of item (ie. just searching for "blueprint"). I don't mind the delay of the free site or the fact that it would take me several attempts to get the information I need. I'm in no rush and if I only get a handful of results instead of a database of hundreds of results... that's just fine.

    I don't care about PVP stats or player kills or any of that kind of stuff. Just some simple searches on where a few specific things were/are found in the game.

    So, in my case, I'm just saying I don't feel like it's worth it to pay for a service/tool that provides more than I need, especially when all I need is possible for me from a free to use website... or ... to get myself once I figure out how to do it. :D
     
    Rowell, Net, Aeryk and 1 other person like this.
  2. Undone

    Undone Trap Master Moderator SOTA Developer

    Messages:
    344
    Likes Received:
    837
    Trophy Points:
    43
    Complaining?! =O
     
    Titania Xylia, Jaesun, Aeryk and 2 others like this.
  3. devilcult

    devilcult Avatar

    Messages:
    901
    Likes Received:
    1,920
    Trophy Points:
    93
    this complaint ;p
     
    Titania Xylia likes this.
  4. Undone

    Undone Trap Master Moderator SOTA Developer

    Messages:
    344
    Likes Received:
    837
    Trophy Points:
    43
    Looks like rational statements/requests to me =)
     
    Titania Xylia, Net, Sentinel2 and 2 others like this.
  5. Aeryk

    Aeryk Avatar

    Messages:
    109
    Likes Received:
    141
    Trophy Points:
    18
    Location:
    Terra (thus far)
    That is not a complaint. He is saying a small number of people are hammering ElasticSearch and if these people do not throttle their usage Portalium may need to shut down our external access to it.

    What they have provided is not optimized in any way for massive query/user loads. Setting up a proper API (likely to include some sort of caching to keep costs down) is an incredibly heavy and ongoing software engineering task. The groups that are making money from this (though i doubt this is very much money) should be particularly conscientious since they are using a free resource.

    This said, there is most definitely value in caching the data (all-be-it at the cost of significantly more complexity). Any set of applications that naively polls the ElasticSearch instance on behalf of thousands of potential users will perform far worse (remember the 5 second limit is IP based). Back to the first statement though, setting up more resilient and scale-able APIs is HARD engineering. We know who these people are, so let's just have a conversation about what Portalium can handle. I have a couple decades of experience with these types of issues, so let me know how i can help.

    Edit to add: The data amounts we are talking about here are actually quite small compared to other modern systems. There are brute force ways to scale Lucene if a group was able to take on the associated cost for clusters/shards. Ultimately though, modeling the data and its access is usually (not always!) more cost efficient for longer lasting initiatives.
     
    Last edited: Oct 31, 2018
    Sentinel2 and the Lacedaemonian like this.
  6. Tiina Onir

    Tiina Onir Avatar

    Messages:
    991
    Likes Received:
    1,724
    Trophy Points:
    105
    Location:
    Bramble, South Paladis
    I don't think anyone was saying the complaint wasn't JUSTIFIED. Clearly, you shouldn't be calling the API every time a user loads your page. Thus far it seems that @Undone just pointing that out has been enough (or I assume it has, as he hasn't brought it up again). When it came up originally (if memory serves) there was discussion about making us register for a key to access the API. If it becomes a problem again, that remains an option.

    Why are we even discussing this?
     
    Thesisus likes this.
  7. Undone

    Undone Trap Master Moderator SOTA Developer

    Messages:
    344
    Likes Received:
    837
    Trophy Points:
    43
    btw I was just teasing @devilcult ; the api rush has passed and everything is cool now. right? right?!
     
  8. devilcult

    devilcult Avatar

    Messages:
    901
    Likes Received:
    1,920
    Trophy Points:
    93
    lol calm down everyone ;p im french and maybe the word was too harsh, but its the only one i know to refer to that, i know you were teasing ;p
    nothing is wrong hehe

    all this to say that was the way people got the entire list of stuff at beginning, now if you want it again, you might hammer the server but undone might "not like it" without complaining ;p
     
    Titania Xylia, Rentier, Aeryk and 2 others like this.
  9. Rafael

    Rafael Avatar

    Messages:
    3
    Likes Received:
    13
    Trophy Points:
    3
    Is there a way to map the SceneName to a more recognizable place name? For example, where is "POT_forest_metropolis_01_template?" @Undone

    [​IMG]
     
    .dao. and Rentier like this.
  10. Hemswal The Descended

    Hemswal The Descended Avatar

    Messages:
    731
    Likes Received:
    1,516
    Trophy Points:
    93
    POT names are hidden on purpose.
     
  11. Undone

    Undone Trap Master Moderator SOTA Developer

    Messages:
    344
    Likes Received:
    837
    Trophy Points:
    43
    easiest way is to look at the tab delimited file the map uses. And hemswal is correct that the POT names are masked on purpose.
     
  12. Feeyo

    Feeyo Avatar

    Messages:
    1,624
    Likes Received:
    2,399
    Trophy Points:
    113
    Location:
    Aelasar’s Forest
    @Undone The public API is nice! But have you checked out the websocket plugin option for elasticsearch? I really recommend playing around with websocket and or SocketIO.
    Subscribing to a websocket as a client is much more friendly for us and also at the end for your API server. The load and performance will be much better.
     
    Aeryk, Tiina Onir, Sentinel2 and 2 others like this.
  13. Barugon

    Barugon Avatar

    Messages:
    11,668
    Likes Received:
    18,591
    Trophy Points:
    153
    Gender:
    Male
    @Undone
    How about an API to search player vendors (opt-in only and no prices, of course)?
     
    Aeryk and FrostII like this.
  14. FrostII

    FrostII Avatar

    Messages:
    4,498
    Likes Received:
    9,195
    Trophy Points:
    153
    Gender:
    Male
    Location:
    Pacific Northwest
    Who wouldn't "opt-in" to list their vendor ? I mean, you got him there to sell, right ?
    No prices ?
    Hmmm, I would prefer them - but I can understand how people who buy/resell wouldn't be a fan.
    Too bad, IMO.........
     
  15. Titania Xylia

    Titania Xylia Avatar

    Messages:
    340
    Likes Received:
    1,020
    Trophy Points:
    43
    Gender:
    Male
    Are fishing events being tracked in the system? Is it just falling under loot? It would be nice if it was under its own LocationEvent type, so it's easy to segment out.

    Thanks!
     
    FrostII and Echondas like this.
  16. devilcult

    devilcult Avatar

    Messages:
    901
    Likes Received:
    1,920
    Trophy Points:
    93
    its probably under the loot locationevent with a special name on the item like Fish_
    which can make it easier to get, unless the name convention went all nuts and only the name of the fish show up, i didnt check
     
    Titania Xylia likes this.
  17. Aaron Pope

    Aaron Pope Avatar

    Messages:
    2
    Likes Received:
    0
    Trophy Points:
    1
    Does the API still work?
    I am getting 503 Service Unavailable.
    * Connected to shroudoftheavatar.com (54.155.190.206) port 9200 (#0)
    No server is available to handle this request.
     
  18. Titania Xylia

    Titania Xylia Avatar

    Messages:
    340
    Likes Received:
    1,020
    Trophy Points:
    43
    Gender:
    Male
    It could be up today, but Chris mentioned no later than next week on Markee's stream. I'm sure life might get in the way, but it sounds like the task is in the queue to get done.
     
    FrostII likes this.
  19. Undone

    Undone Trap Master Moderator SOTA Developer

    Messages:
    344
    Likes Received:
    837
    Trophy Points:
    43
    I hope to have this up early next week. There are still a few details to work out after the move to the new data center.
     
    Aeryk, Sentinel2, Jaesun and 4 others like this.
  20. Hemswal The Descended

    Hemswal The Descended Avatar

    Messages:
    731
    Likes Received:
    1,516
    Trophy Points:
    93
    @Titania Xylia

    Fishing tab isn't done yet, but to give you an idea:

    [​IMG]
     
    Aeryk likes this.