1. This forum section contains Blog style threads from the Developers, and/or threads with details about what they have been working on, previews of upcoming content, and sometimes backstory associated with the content. Players are permitted to reply to these threads if the developer has indicated it is permissible by leaving the thread unlocked. Please note that TOS applies, and any abuse or trolling will be moderated.
    Dismiss Notice

Public Stats API

Discussion in 'Developer Work and Blog posts' started by Undone, May 9, 2017.

Thread Status:
Not open for further replies.
Page 5 of 26
  1. Umuri

    Umuri Avatar

    Messages:
    527
    Likes Received:
    1,828
    Trophy Points:
    63
    Correct, it will no longer function (as it was intended anyway) due to some changes undone made that removes some of the more troubelsome data pairs i was worried about. So we should thank Undone for his prompt response to this!

    I will still remake it tomorrow probably to be more of an heatmap of player activity of sorts, but that runs into data limits.
     
    Umuri, May 10, 2017
    #81
  2. Ben_Hroth

    Ben_Hroth Avatar

    Messages:
    261
    Likes Received:
    739
    Trophy Points:
    28
    Thanks Umuri, I love SOTA/Stats already! :) Quick question, what does EGD stand for?

    EDIT: Oops, EconomyGoldDelta. Got it!
     
    Ben_Hroth, May 11, 2017
    #82
  3. Umuri

    Umuri Avatar

    Messages:
    527
    Likes Received:
    1,828
    Trophy Points:
    63
    Alright well guys, i need some sleep, but here's some quick bashups for you all:

    http://umuri.com/SOTA/Stats/Top25/

    Including Such Novelties As:
    24HR-CotosConsumedByCrownMerchants
    24HR-DeadliestMonster
    24HR-ItemsBoughtFromMerchant
    24HR-ItemsSoldToMerchant
    24HR-MonstersKilled
    24HR-MostMurdered
    24HR-MostMurders
    24HR-Suicides
    ALL-CotosConsumedByCrownMerchant
    ALL-DeadliestMonster
    ALL-ItemsBoughtFromMerchant
    ALL-ItemsSoldToMerchant
    ALL-MonstersKilled
    ALL-MostMurdered
    ALL-MostMurders
    ALL-Suicides
    LeastCraftedItems
    MostBoughtCrownMerchant
    MostCraftedItems
    MostCraftingDestroyed
    MostCraftingDestroyedEquipable
    MostCraftingMade
    MostCraftingMadeEquipable
    MostLootedWorldItems
    PlacesToGetStuck
     
    Umuri, May 11, 2017
    #83
    Kabalyero, Lord Andernut, hroth and 4 others like this.
  4. that_shawn_guy

    that_shawn_guy Bug Hunter

    Messages:
    1,415
    Likes Received:
    3,753
    Trophy Points:
    125
    Location:
    earth... mostly
    Did something change about the timestamps recently?

    Yesterday I was that seemed to reflect the time of the event. The recent data seems to all have the same time at the exact top of the hour.
     
    that_shawn_guy, May 11, 2017
    #84
  5. Chris

    Chris Tech Lord Moderator Ambassador SOTA Developer

    Messages:
    2,470
    Likes Received:
    27,551
    Trophy Points:
    190
    Gender:
    Male
    Interesting... now I need to go see where people are using stuck in The Rise....
     
    Chris, May 11, 2017
    #85
    Astor Cerberus, hroth, Alley Oop and 1 other person like this.
  6. Wintermute of CoF

    Wintermute of CoF Avatar

    Messages:
    1,372
    Likes Received:
    2,432
    Trophy Points:
    113
    Gender:
    Male
    Location:
    London, United Kingdom
    Given the other places near the top of that list I'm guessing that /stuck is quicker for getting out than going to the menu and selecting start on world map.
     
    Wintermute of CoF, May 11, 2017
    #86
  7. Undone

    Undone Trap Master Moderator SOTA Developer

    Messages:
    346
    Likes Received:
    848
    Trophy Points:
    43
    Right now all POTs are technically anonymous since the SceneName field only shows the base scene name (or template).

    I believe you are correct.

    You are welcome! These are things we may be able to add in the future but I can't promise any time tables.
     
    Undone, May 11, 2017
    #87
    Ancev likes this.
  8. Undone

    Undone Trap Master Moderator SOTA Developer

    Messages:
    346
    Likes Received:
    848
    Trophy Points:
    43
    It should be immediate. Or near immediate.
    I should be able to do that. Which are the most confusing and I'll focus on hunting those down first.
    Glad I could help make sense of it!
    We'll have to have some internal discussion on this; but even if we did this it might be a while before it was implemented.
    External kibanas are likely to not work due to limitations imposed by our security measures.
     
    Undone, May 11, 2017
    #88
    Kabalyero, scroda and Ancev like this.
  9. Undone

    Undone Trap Master Moderator SOTA Developer

    Messages:
    346
    Likes Received:
    848
    Trophy Points:
    43
    Yes. I stripped minutes & seconds from the logs so things would be a bit more difficult to correlate. A bit too much of the mystery is taken away if you know all the loot tables =O
     
    Undone, May 11, 2017
    #89
    Duke Gréagóir, Umuri and devilcult like this.
  10. that_shawn_guy

    that_shawn_guy Bug Hunter

    Messages:
    1,415
    Likes Received:
    3,753
    Trophy Points:
    125
    Location:
    earth... mostly
    Can we add it back for the ItemGained_Merchant and other merchant related events?

    Thank you
     
    that_shawn_guy, May 11, 2017
    #90
  11. that_shawn_guy

    that_shawn_guy Bug Hunter

    Messages:
    1,415
    Likes Received:
    3,753
    Trophy Points:
    125
    Location:
    earth... mostly
    Are all vendors included in the data or is just a subset?

    So far, I've not found any of the test purchase I've made.

    Thank you
     
    that_shawn_guy, May 11, 2017
    #91
  12. tuerdkughter

    tuerdkughter Avatar

    Messages:
    6
    Likes Received:
    2
    Trophy Points:
    3
    @Undone Can put the accurate timestamps back in? Obscuring the timestamps seems like a silly hack, not a real security control anyway. I'd like to use this data to make a real-time map under Linux. Since the Linux version of the in-game map doesn't work.
     
    tuerdkughter, May 11, 2017
    #92
    that_shawn_guy likes this.
  13. Undone

    Undone Trap Master Moderator SOTA Developer

    Messages:
    346
    Likes Received:
    848
    Trophy Points:
    43
    Not at the moment. Will talk about this more with the team early next week.
    The economy logging is a bit trickier and still being investigated.
    Not right now. It leaves open too many doors that we prefer to keep closed. I'm still working on a linux sotamap port but CEF is not playing nice.
     
    Undone, May 11, 2017
    #93
    Duke Gréagóir likes this.
  14. DavidDC

    DavidDC Programmer Moderator SOTA Developer

    Messages:
    1,532
    Likes Received:
    3,236
    Trophy Points:
    113
    Gender:
    Male
    Seem like i can only use the get method to use the elasticsearch from nodejs.. But i cannot use a body with the get method so if i want to add a body i gotta do a post which give me that forbidden 403 error authorization. Also tryed using many elasticsearch explorer client and cant do nothing with that. I dont see anything about the get method in ur python script and i dont understand how you are accessing it exactly. Maybe its my body that isnt right and server is refusing me? Can someone check up what i am doing wrong?

    Code:
    var elasticsearch = require('elasticsearch');
//var json2html = require('node-json2html');

var client = new elasticsearch.Client({
  host: 'shroudoftheavatar.com:9200',
  log: 'trace'
});

//client.cluster.health({},function(err,resp,status) {
//  console.log("-- Client Health --",resp);
//});

client.search({
  //method: 'get',
  index: "logstash-2017.05.11",
  body: {
    "query_string": {
        "query": "LootGenerated",
        "analyze_wildcard": true,
    }
  }
}).then(function (resp) {
    var hits = resp.hits.hits;
    console.log("Got response in " + resp.took + " milliseconds");
    console.log(hits);
}, function (err) {
    console.trace(err.message);
});
    This is my error message:

    Code:
    Elasticsearch INFO: 2017-05-12T00:12:52Z
  Adding connection to http://shroudoftheavatar.com:9200/

Elasticsearch DEBUG: 2017-05-12T00:12:52Z
  starting request {
    "method": "POST",
    "path": "/logstash-2017.05.11/_search",
    "body": {
      "query_string": {
        "query": "LootGenerated",
        "analyze_wildcard": true
      }
    },
    "query": {}
  }


Elasticsearch TRACE: 2017-05-12T00:12:52Z
  -> POST http://shroudoftheavatar.com:9200/logstash-2017.05.11/_search
  {
    "query_string": {
      "query": "LootGenerated",
      "analyze_wildcard": true
    }
  }
  <- 403
  <html><body><h1>403 Forbidden</h1>
     
    Last edited: May 11, 2017
    DavidDC, May 11, 2017
    #94
  15. Schemer

    Schemer Avatar

    Messages:
    5
    Likes Received:
    8
    Trophy Points:
    3
    @devilcult I got around this same issue yesterday. You have to use GET and the query string parameters. I'll link a gist to the example code in a bit.
     
    Schemer, May 11, 2017
    #95
  16. DavidDC

    DavidDC Programmer Moderator SOTA Developer

    Messages:
    1,532
    Likes Received:
    3,236
    Trophy Points:
    113
    Gender:
    Male
    Yeah i got it working like this... Just i gotta put all on same query line. I dont know how to put timestamp and all in that

    Code:
    client.search({
  method: 'get',
  index: "*",
  q: 'LootGenerated SceneName=Rise'
}).then(function (resp) {
    var hits = resp.hits.hits;
    console.log("Got response in " + resp.took + " milliseconds");
    console.log(hits);
}, function (err) {
    console.trace(err.message);
});
     
    DavidDC, May 11, 2017
    #96
  17. Lained

    Lained Avatar

    Messages:
    2,803
    Likes Received:
    4,794
    Trophy Points:
    165
    Location:
    Yeovil, England
    HTML:
    <!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>SotA</title>
<script type='text/javascript' src="https://code.jquery.com/jquery-3.2.1.min.js"></script>
<script>

    var url = "http://shroudoftheavatar.com:9200/_all/_search?"

    function getData() {

        $.ajax({

            url: url + encodeURI($(':input[name=query]').val()),
            dataType: "text",
            beforeSend: function(){
                $('.result').text('')
            },

        }).done(function(responseText) {

          $('.result').text(JSON.stringify(JSON.parse(responseText), null, '\t'));

        });

        return false
    }



</script>
</head>

<body>
<form onsubmit="return getData()">
<input type="text" name="query" style="width: 90%" value="size=100&q=LocationEvent:ItemGained_Loot&_source=Archetype,Quantity,EconomyGoldDelta,SceneName" />
<input type="submit" value="Retrieve Data" />
</form>
<pre class="result"></pre>
</body>
</html>
     
    Lained, May 11, 2017
    #97
    Browncoat Jayson, Kabalyero and devilcult like this.
  18. DavidDC

    DavidDC Programmer Moderator SOTA Developer

    Messages:
    1,532
    Likes Received:
    3,236
    Trophy Points:
    113
    Gender:
    Male
    With that line size=100&q=LocationEvent:ItemGained_Loot&_source=Archetype,Quantity,EconomyGoldDelta,SceneName
    What if i want to check everything that dropped only today? How could i set the timestamp to search only from there and then
     
    DavidDC, May 11, 2017
    #98
  19. Undone

    Undone Trap Master Moderator SOTA Developer

    Messages:
    346
    Likes Received:
    848
    Trophy Points:
    43
    only GET calls are allowed. I'm building the body of my query in utils/query_builder.py
     
    Undone, May 11, 2017
    #99
  20. DavidDC

    DavidDC Programmer Moderator SOTA Developer

    Messages:
    1,532
    Likes Received:
    3,236
    Trophy Points:
    113
    Gender:
    Male
    If anyone know how i can pass this json format to the query using node js and the method get please lemme know...

    Code:
    var elasticsearch = require('elasticsearch');
//var json2html = require('node-json2html');

var client = new elasticsearch.Client({
  host: 'shroudoftheavatar.com:9200',
  log: 'trace'
});

var REQUEST = {
    "size": 5,
    "sort": [
        {
            "@timestamp": {
                "order": "desc",
                "unmapped_type": "boolean"
            }
        }
    ],
    "query": {
        "bool": {
            "must": [
                {
                    "query_string": { "query": "ItemGained_Loot&SceneName=Rise", "analyze_wildcard": true }
                },
                {
                    "range": {
                        "@timestamp": {
                            "gte": 1494460800000,
                            "lte": 1494556601706,
                            "format": "epoch_millis"
                        }
                    }
                }
            ],
            "must_not": []
        }
    },
}
REQUEST = JSON.stringify(REQUEST);
//console.log(JSON.stringify(REQUEST));
client.search({
  method: 'get',
  index: "*",
  q: REQUEST
}).then(function (resp) {
    var hits = resp.hits.hits;
    console.log("Found " + resp.counts + " in " + resp.took + " milliseconds");
    console.log(hits);
}, function (err) {
    console.log(REQUEST);
    console.trace(err.message);
});

    Trace error message:

    Code:
    C:\Users\DevilCult\Desktop>node test
Elasticsearch INFO: 2017-05-12T03:00:51Z
  Adding connection to http://shroudoftheavatar.com:9200/

Elasticsearch DEBUG: 2017-05-12T03:00:51Z
  starting request {
    "method": "GET",
    "path": "/*/_search",
    "query": {
      "q": "{\"size\":5,\"sort\":[{\"@timestamp\":{\"order\":\"desc\",\"unmapped_type\":\"boolean\"}}],\"query\":{\"bool\":{\"must\":[{\"query_string\":{\"query\":\"ItemGained_Loot&SceneName=Rise\",\"analyze_wildcard\":true}},{\"range\":{\"@timestamp\":{\"gte\":1494460800000,\"lte\":1494556601706,\"format\":\"epoch_millis\"}}}],\"must_not\":[]}}}"
    }
  }


Elasticsearch TRACE: 2017-05-12T03:00:55Z
  -> GET http://shroudoftheavatar.com:9200/*/_search?q=%7B%22size%22%3A5%2C%22sort%22%3A%5B%7B%22%40timestamp%22%3A%7B%22order%22%3A%22desc%22%2C%22unmapped_type%22%3A%22boolean%22%7D%7D%5D%2C%22query%22%3A%7B%22bool%22%3A%7B%22must%22%3A%5B%7B%22query_string%22%3A%7B%22query%22%3A%22ItemGained_Loot%26SceneName%3DRise%22%2C%22analyze_wildcard%22%3Atrue%7D%7D%2C%7B%22range%22%3A%7B%22%40timestamp%22%3A%7B%22gte%22%3A1494460800000%2C%22lte%22%3A1494556601706%2C%22format%22%3A%22epoch_millis%22%7D%7D%7D%5D%2C%22must_not%22%3A%5B%5D%7D%7D%7D

  <- 400
  {
    "error": {
      "root_cause": [
        {
          "type": "query_shard_exception",
          "reason": "Failed to parse query [{\"size\":5,\"sort\":[{\"@timestamp\":{\"order\":\"desc\",\"unmapped_type\":\"boolean\"}}],\"query\":{\"bool\":{\"must\":[{\"query_string\":{\"query\":\"ItemGained_Loot&SceneName=Rise\",\"analyze_wildcard\":true}},{\"range\":{\"@timestamp\":{\"gte\":1494460800000,\"lte\":1494556601706,\"format\":\"epoch_millis\"}}}],\"must_not\":[]}}}]",
          "index_uuid": "_CHzKxlKSL2tdgqhxhV3tg",
          "index": "logstash-2017.04.27"
        },
     
    DavidDC, May 11, 2017
    #100
Page 5 of 26
Thread Status:
Not open for further replies.