1. This forum section contains Blog style threads from the Developers, and/or threads with details about what they have been working on, previews of upcoming content, and sometimes backstory associated with the content. Players are permitted to reply to these threads if the developer has indicated it is permissible by leaving the thread unlocked. Please note that TOS applies, and any abuse or trolling will be moderated.
    Dismiss Notice

Public Stats API

Discussion in 'Developer Work and Blog posts' started by Undone, May 9, 2017.

Thread Status:
Not open for further replies.
  1. Phil Scott

    Phil Scott Avatar

    Messages:
    74
    Likes Received:
    41
    Trophy Points:
    20
    Gender:
    Male
    ItemGained_Merchant is only buys from NPC Merchants (not Crown they are a different list)
    I have the complete list of those Archetypes in database that generates The Drover's Road Journal and could expose that if people need.
    (As we don't have categories nor item names yet I use the Archetype to generate the name of the items for now).

    Update:
    I have update The Drover's Road Journal Items page to include the Archetype as a hidden column. If you go to the Items Page and click on the columns button you will see "Archetype", click on that and your browser will update to show the Archetype column. You can shift the columns around with drag and drop, change what columns are shown from the column button, sort up or down by clicking the column header and filter with the search box (even hidden columns are searched), and export to CSV with the CSV button. I hope this is useful.
     
    Last edited: Jun 4, 2017
  2. Ben_Hroth

    Ben_Hroth Avatar

    Messages:
    261
    Likes Received:
    739
    Trophy Points:
    28
    Has anyone found the Archetype for the Cabalist Hoods? I saw their weapons, but not their hoods.

    EDIT: Sorry, just found them. They were under Clothing.
     
    Last edited: Jun 5, 2017
  3. Arkah EMPstrike

    Arkah EMPstrike Avatar

    Messages:
    4,542
    Likes Received:
    8,100
    Trophy Points:
    153
    im diggin the drovers road journal. Well done phil! Very useful tool for economic analysis
     
  4. Dereck Visaard

    Dereck Visaard Avatar

    Messages:
    252
    Likes Received:
    444
    Trophy Points:
    40
    Gender:
    Male
    Location:
    Austin, TX
    Thanks Phil. The list is very helpful. @Undone, is there an ETA on when player-owned merchants in NPC towns will be included?
     
  5. DavidDC

    DavidDC Programmer Moderator SOTA Developer

    Messages:
    1,532
    Likes Received:
    3,236
    Trophy Points:
    113
    Gender:
    Male
    Id also like to know when compression will be enabled to be sure its working on my side lol :p i think im set to receive it... not sure its done on es server yet
     
  6. Undone

    Undone Trap Master Moderator SOTA Developer

    Messages:
    346
    Likes Received:
    848
    Trophy Points:
    43
    No eta as of yet.
    From what I can tell http compression is on by default; I haven't tried it myself but am curious how large of a difference it makes for you.
     
  7. DavidDC

    DavidDC Programmer Moderator SOTA Developer

    Messages:
    1,532
    Likes Received:
    3,236
    Trophy Points:
    113
    Gender:
    Male
    Ok i figured out what was wrong... i think compression from my nodejs server to ur ES DB is compressed by default as i could read from es server and my nodejs ES api, the problem is my nodejs server to javascript client which send raw data which was sending wrong value:p

    So after a few test with wireshark checking datas in (cause there is no way i can tell with js apis cause compression is automatic and when i check its already raw)...
    a 2.71mb data became 442kbs

    Just to add a little to the stats from my app since about 2 weeks
    2352 request for about 370mb, reduced by about 80% cause of compression is about 70mbs total used :p
    (i wonder who the heck did that 30gb transfert that day... lol its crazy)

    Btw As i am using a server to retrieve data for multiple client, if multiple client ask data at the same time are all request going to be accepted? Whats the limit?
     
    Last edited: Jun 7, 2017
  8. Jezebel Caerndow

    Jezebel Caerndow Avatar

    Messages:
    3,118
    Likes Received:
    7,912
    Trophy Points:
    153
    This is all french to me, and I am Canadian and that is suppose to be my second language :). Must be why devil understands it LOL.
     
    King Robert and devilcult like this.
  9. DavidDC

    DavidDC Programmer Moderator SOTA Developer

    Messages:
    1,532
    Likes Received:
    3,236
    Trophy Points:
    113
    Gender:
    Male
    Hmm i wanted to create a "map activity" image with a map and where the action is happening, started with deep ravenswood center of map was 0,0 (xpos,zpos) all good... went to southern grunvald desert to test on that map since its easier to know where people is and to figure out if my dots on the map are correctly placed, but found out that 0,0 is top right of that map.... No map is starting at the same position? o_O

    I wonder if all the map are different like that or only a few old map wasnt 'correctly' set up...
     
    2112Starman likes this.
  10. Wintermute of CoF

    Wintermute of CoF Avatar

    Messages:
    1,372
    Likes Received:
    2,432
    Trophy Points:
    113
    Gender:
    Male
    Location:
    London, United Kingdom
    Might be worth comparing Braemar and Soltown to see how the 0,0 point compares, since the latter started out as a rotated clone of the former?
     
  11. DavidDC

    DavidDC Programmer Moderator SOTA Developer

    Messages:
    1,532
    Likes Received:
    3,236
    Trophy Points:
    113
    Gender:
    Male
    What is this locationevent mean? MonsterLeashed
     
    2112Starman likes this.
  12. Undone

    Undone Trap Master Moderator SOTA Developer

    Messages:
    346
    Likes Received:
    848
    Trophy Points:
    43
    It's an internal event I should remove. Thanks for bringing it up!
     
  13. DavidDC

    DavidDC Programmer Moderator SOTA Developer

    Messages:
    1,532
    Likes Received:
    3,236
    Trophy Points:
    113
    Gender:
    Male
    trying to do some aggregations and im getting an error "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [EconomyGoldDelta] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory."

    What am i doing wrong?
    Code:
    "size": 10,
            "sort": [
                {
                  "@timestamp": {
                        "order": "desc",
                        "unmapped_type": "boolean"
                  }
                }
            ],
            "query": {
                "bool": {
                  "must": [
                        {
                          "query_string": {
                                "query": "Archetype: 'ItemRawMat_Ore_Silver' AND LocationEvent: 'ItemGained_Loot'",
                                "analyze_wildcard": "true"
                          }
                        },
                        {
                          "range": {
                                "@timestamp": {
                                  "gte": gte_stamp,
                                  "lte": lte_stamp,
                                  "format": "epoch_millis"
                                }
                          }
                        }
                  ],
                  "must_not": []
                }
            },
            "aggs": {
                "matrixstats": {
                    "matrix_stats": {
                        "fields": ["GoldValue", "EconomyGoldDelta"]
                    }
                }
            }

    simple one like
    Code:
    "aggs" : {
       "avg_grade" : { "avg" : { "field" : "GoldValue" } }
    }

    Doesnt work either... same error
    Code:
     {
        "took": 1213,
        "timed_out": false,
        "_shards": {
          "total": 250,
          "successful": 105,
          "failed": 145,
          "failures": [
            {
              "shard": 0,
              "index": "logstash-2017.05.18",
              "node": "ylXWkgiDRtK0M14dRHmKXg",
              "reason": {
                "type": "illegal_argument_exception",
                "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [GoldValue] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory."
              }
            }
          ]
        },
        "hits": {
          "total": 0,
          "max_score": null,
          "hits": []
        },
        "aggregations": {
          "avg_grade": {
            "value": null
          }
        }
      }
     
  14. Undone

    Undone Trap Master Moderator SOTA Developer

    Messages:
    346
    Likes Received:
    848
    Trophy Points:
    43
    Dynamic scripting should be disabled so you'll have to do any aggregations on your end.
     
  15. Echondas

    Echondas Bug Hunter Bug Moderator

    Messages:
    3,784
    Likes Received:
    3,996
    Trophy Points:
    165
    Gender:
    Male
    Location:
    NY
    FYI - seeing errors from @Umuri 's page here: http://umuri.com/SOTA/Stats/Top25/
    Examples are:
    Statistics Error: Table 'sotaapi.PlayerKilledBySelf' doesn't exist
    Statistics Error: Table 'sotaapi.ItemGained_Loot' doesn't exist
    Statistics Error: Table 'sotaapi.MonsterKilledByPlayer' doesn't exist
    ..
     
  16. Umuri

    Umuri Avatar

    Messages:
    527
    Likes Received:
    1,828
    Trophy Points:
    63
    It's been taken offline in prep for the launch of SotaSTATS, hopefully in a few days. It was mostly defunct at this point anyway, it hadn't had its dataloader running for weeks.
     
    Echondas likes this.
  17. Magnus Zarwaddim

    Magnus Zarwaddim Avatar

    Messages:
    975
    Likes Received:
    1,884
    Trophy Points:
    93
    I am curious if you are able to perform searches based on mobs and their density. Yes, shameless as I am, I want to look up the best place to farm things. You can do this with loot; you can also get an aggregate of player kills of the types of mobs.
     
    Last edited: Jun 22, 2017
  18. Phil Scott

    Phil Scott Avatar

    Messages:
    74
    Likes Received:
    41
    Trophy Points:
    20
    Gender:
    Male
    You would have to track loot and how many player are in a map. The issue is that number of players, but you can get that from PositionUpdate. I don't collect how many people are in a zone for The Drover's Road Journal, that would be data that I don't need for anything else but it could be done with a simple script and a quick data base. The DRJ does have the per day count of loot dropped for
    each Archetype.

    OOPS I did not look at the quick data check that I did carefully enough. PositionUpdate was take out on 15 May at 1100 and is no longer on the interface, there is no way to figure out how many people are in a zone at any one time therefor you can't tell if a zone is dropping lots of stuff or just has a lot of people in it.
     
    Last edited: Jun 23, 2017
  19. Magnus Zarwaddim

    Magnus Zarwaddim Avatar

    Messages:
    975
    Likes Received:
    1,884
    Trophy Points:
    93
    Thanks, it makes sense to "follow the loot" as what I had in mind was finding the best places to grind, for example, obsidian chips.
     
  20. that_shawn_guy

    that_shawn_guy Bug Hunter

    Messages:
    1,415
    Likes Received:
    3,753
    Trophy Points:
    125
    Location:
    earth... mostly
    Since there was nothing about the Public Stats API in the release notes, is it safe to assume there were no changes to the Public Stats API in R43?

    Could @Undone or another dev clarify the intention of providing all vendor data?

    Thank you
     
Thread Status:
Not open for further replies.