1. Before posting any MODs, please contact us at support@portalarium.com to ensure your MOD is acceptable to the Terms of Service and EULA of the game. Portalarium is not responsible for use of MODs and possible adverse effects on your client or computer, and we may take action on accounts using MODs that are deemed illegal. Whether submitted here or elsewhere in the community, use MODs at your own risk!
    Dismiss Notice

Umuri's SotaSTATS - Making the API Accessible to all!

Discussion in 'Player Created Resources' started by Umuri, Sep 18, 2017.

  1. Umuri

    Umuri Avatar

    Messages:
    529
    Likes Received:
    1,874
    Trophy Points:
    63
    Alright, after way too many delays and me dropping off the map for over a month past when it should have deployed, i'd like to announce the 1.0 version of SotaSTATS is released.

    SotaSTATS

    All the main features I wanted for 1.0 work, to a decent degree of speed.
    Things you will find here:
    Raw API Data -> Find under codex. This is the slowest feature of the sight, but can be used to pull any raw data the API has provided.
    Overlayed API data on maps -> Find under codex once you've specified a date range (max 48 hours) a location, and at least an event type. There will be a link on the left saying Overlay Codex data. Example
    Statistics -> This will give you more refined statistics based on 4 main categories:
    Monsters - Data on monsters, what they drop, where they spawn, as people populate them hp/etc.
    Items - Where you can get them, what they are, destroy/creation totals over the past few days
    Scenes - What appears there (monsters/items)
    Misc - Lottery Stats, Coto usage stats

    And more to be found as you explore around.
    If you find something you want to share, just post the link. SotaSTATS tries to ensure the links contain all needed information, to show others what you see.

    In addition, I would like to announce that SotaSTATS uses a demo of a new login tech i've been working on.
    SotaSTATS requires NO SIGN UP. But if you want to login, you can login IN-GAME, without ever revealing anything other than your character name. This new login method will allow us to make guild websites/auction houses/events/etc and have a provable method of limiting it to 1 per in-game account. As well as other nifty features. You don't have to trust me with passwords, usernames, etc. It's all handled in-game on portalarium's servers!

    Now, once you login, you'll get access to more features such as more codex entries returned on searches, access to the list of what you've killed over the past 30 days.
    More importantly, once you login you can start submitting updates to the SotaSTATS internal library.
    Right now this is only monsters. To do this, go to the stats tab, pull up the monster, then go to the specific scene. You'll see its stats on the left (hp/focus/exp/etc). You can submit edits to that if you are logged in, that help update data for others.

    I will be expanding editing abilities to include names (so we can use better names other than the rediculous internal names the game uses).

    Oh yeah and i tried to get it to not look entirely bad which is my trademark, and somewhat play nice on mobile, so if you have mobile complaints, let me hear it!

    ONE MORE THING: The loot tables are estimates and guesses, and should not be taken as word of god. They have lots of false positives. I just thought throwing something together would be better than nothing, as Atos has gone out of his way (rightly so) to make matching up loot to kills difficult. :p They'll get better as I iterately progress the algorithm that matches up.

    So yeah, have fun, more updates to come.
    I WILL be fixing speed over the next few days and watching for any big performance hangups through real usage.
     
    Last edited: Sep 19, 2017
    Sentinel2, Jaesun, Womby and 12 others like this.
  2. Balandar

    Balandar Avatar

    Messages:
    412
    Likes Received:
    1,087
    Trophy Points:
    55
    Gender:
    Male
    Location:
    Honor Hold
    Nice work :)
     
  3. LoneStranger

    LoneStranger Avatar

    Messages:
    3,047
    Likes Received:
    4,773
    Trophy Points:
    165
    Gender:
    Male
    Location:
    Petaluma, CA
    Great job!

    I notice that the translated names are being returned as individual objects/mobs. I can only imagine that would affect overall stats by splitting things up by translated name.
     
  4. Umuri

    Umuri Avatar

    Messages:
    529
    Likes Received:
    1,874
    Trophy Points:
    63
    It does, however I already have a system in-place to merge them and soft-link different names as translations of a main base object.
    However, this requires people flagging what is translations of what. An example of this is the Ferocious Red Spider
    Example

    I have not made an interface for this, as it's a one-off easy to do job, that I may hire-out to players for in-game gold.
     
    LoneStranger likes this.
  5. LoneStranger

    LoneStranger Avatar

    Messages:
    3,047
    Likes Received:
    4,773
    Trophy Points:
    165
    Gender:
    Male
    Location:
    Petaluma, CA
    I wonder if there is a set of internal resource files that could be parsed into a database to get it all in one-fell-swoop.
     
  6. Umuri

    Umuri Avatar

    Messages:
    529
    Likes Received:
    1,874
    Trophy Points:
    63
    Monday - 7:31 pm - Well you all hit it enough that the host decided it was "too busy" and killed it without contacting me.
    I am very pissed at them and am working to resolve it, hopefully it will be back up within 48 hours.

    Update:
    Wednesday 11:36 am - I have it tenatively back up, having built an entire new server for the dba, migrated over, then doing lots of upgrades to deal with the new architecture since the web and db are now far apart. It'll be probably not until tonight that it's fully caught up to current data, so expect things to be wonky.
     
    Last edited: Sep 27, 2017
    FrostII and Kabalyero like this.
  7. Vladamir Begemot

    Vladamir Begemot Avatar

    Messages:
    3,781
    Likes Received:
    7,214
    Trophy Points:
    153
    Gender:
    Male
    Hi, it feels like the Codex search should be defaulting to the most recent searchable time period instead of the earliest one.

    And is there a list of search terms somewhere? Like all available EventTypes?

    Thanks,

    And this is very exciting, still.
     
    FrostII likes this.
  8. Jens_T

    Jens_T Avatar

    Messages:
    616
    Likes Received:
    1,396
    Trophy Points:
    93
    Location:
    Zurich
    @Umuri in case you’re still developing sotastats would it be possible to add a report on logged in players?
    There’s a lot of discussion on player count and @Chris pointed to the API for this kind of information - but writing the scripts is beyond me.
     
    MrBlight and Lord Andernut like this.
  9. Mykll

    Mykll Avatar

    Messages:
    674
    Likes Received:
    1,229
    Trophy Points:
    105
    Gender:
    Male
    Location:
    Davidian Bunker
    So not just me then? People asking how many people play, people get mad and link to here, yet I can't find current or historical people playing.
     
    MrBlight likes this.
  10. Umuri

    Umuri Avatar

    Messages:
    529
    Likes Received:
    1,874
    Trophy Points:
    63
    Can you provide me a quote/link to where he says this? As far as i know, there is nothing in the API that would easily provide this information without making a few inferences/assumptions.

    I had such a thing that made those when it first come out, but i will say unless somethings changed since i last looked there is no easy/foolproof way to do this to give really solid numbers
    I used the # of people who complete their oracle quest as a "rough" estimate.
    Keep in mind this is not 1000%, I would assign a rough error rate of at most 5% to this number, as some of the things this picks up are quest turn ins that aren't the oracle quest. But the running average can be taken to reduce it to what is "most likely" just the oracle quests.

    This does not tell you how many people play, but merely how many accounts get their oracle quest each day for 500 gold.
    You can make your own arguments on how many accounts per person, how many are bots, how many people play without collecting the oracle each day, etc etc.

    However, below is the requested data, query time 2808 seconds.
    You can interpret the trends yourselves, however the TLDR is that we went from ~800 a day to ~1000, so we could estimate assuming all else constant, we've gained 20% since may 2017.


    2017-04-28 766
    2017-04-29 840
    2017-04-30 806
    2017-05-01 769
    2017-05-02 801
    2017-05-03 723
    2017-05-04 652
    2017-05-05 756
    2017-05-06 796
    2017-05-07 772
    2017-05-08 719
    2017-05-09 725
    2017-05-10 681
    2017-05-11 706
    2017-05-12 747
    2017-05-13 728
    2017-05-14 750
    2017-05-15 713
    2017-05-16 702
    2017-05-17 726
    2017-05-18 733
    2017-05-19 705
    2017-05-20 753
    2017-05-21 737
    2017-05-22 739
    2017-05-23 678
    2017-05-24 703
    2017-05-25 735
    2017-05-26 740
    2017-05-27 744
    2017-05-28 753
    2017-05-29 743
    2017-05-30 696
    2017-05-31 748
    2017-06-01 790
    2017-06-02 770
    2017-06-03 793
    2017-06-04 821
    2017-06-05 726
    2017-06-06 729
    2017-06-07 706
    2017-06-08 731
    2017-06-09 727
    2017-06-10 781
    2017-06-11 814
    2017-06-12 732
    2017-06-13 731
    2017-06-14 739
    2017-06-15 727
    2017-06-16 733
    2017-06-17 728
    2017-06-18 756
    2017-06-19 698
    2017-06-20 705
    2017-06-21 709
    2017-06-22 686
    2017-06-23 708
    2017-06-24 695
    2017-06-25 696
    2017-06-26 681
    2017-06-27 676
    2017-06-28 698
    2017-06-29 785
    2017-06-30 871
    2017-07-01 862
    2017-07-02 808
    2017-07-03 796
    2017-07-04 839
    2017-07-05 773
    2017-07-06 817
    2017-07-07 828
    2017-07-08 880
    2017-07-09 861
    2017-07-10 812
    2017-07-11 833
    2017-07-12 885
    2017-07-13 863
    2017-07-14 833
    2017-07-15 900
    2017-07-16 896
    2017-07-17 826
    2017-07-18 845
    2017-07-19 862
    2017-07-20 831
    2017-07-21 772
    2017-07-22 843
    2017-07-23 884
    2017-07-24 805
    2017-07-25 793
    2017-07-26 799
    2017-07-27 817
    2017-07-28 850
    2017-07-29 902
    2017-07-30 903
    2017-07-31 804
    2017-08-01 835
    2017-08-02 839
    2017-08-03 839
    2017-08-04 822
    2017-08-05 831
    2017-08-06 883
    2017-08-07 858
    2017-08-08 836
    2017-08-09 851
    2017-08-10 846
    2017-08-11 850
    2017-08-12 891
    2017-08-13 900
    2017-08-14 843
    2017-08-15 842
    2017-08-16 901
    2017-08-17 860
    2017-08-18 877
    2017-08-19 883
    2017-08-20 901
    2017-08-21 830
    2017-08-22 858
    2017-08-23 842
    2017-08-24 841
    2017-08-25 840
    2017-08-26 892
    2017-08-27 958
    2017-08-28 803
    2017-08-29 484
    2017-09-01 92
    2017-09-02 952
    2017-09-03 941
    2017-09-04 965
    2017-09-05 855
    2017-09-06 884
    2017-09-07 864
    2017-09-08 897
    2017-09-09 896
    2017-09-10 896
    2017-09-11 865
    2017-09-12 872
    2017-09-13 880
    2017-09-14 911
    2017-09-15 889
    2017-09-16 923
    2017-09-17 926
    2017-09-18 859
    2017-09-19 893
    2017-09-20 861
    2017-09-21 843
    2017-09-22 853
    2017-09-23 914
    2017-09-24 897
    2017-09-25 879
    2017-09-26 848
    2017-09-27 865
    2017-09-28 878
    2017-09-29 866
    2017-09-30 881
    2017-10-01 881
    2017-10-02 861
    2017-10-03 855
    2017-10-04 859
    2017-10-05 872
    2017-10-06 865
    2017-10-07 899
    2017-10-08 920
    2017-10-09 888
    2017-10-10 844
    2017-10-11 894
    2017-10-12 889
    2017-10-13 881
    2017-10-14 887
    2017-10-15 915
    2017-10-16 899
    2017-10-17 875
    2017-10-18 902
    2017-10-19 866
    2017-10-20 865
    2017-10-21 884
    2017-10-22 893
    2017-10-23 860
    2017-10-24 861
    2017-10-25 822
    2017-10-26 852
    2017-10-27 863
    2017-10-28 892
    2017-10-29 880
    2017-10-30 868
    2017-10-31 881
    2017-11-01 878
    2017-11-02 900
    2017-11-03 898
    2017-11-04 941
    2017-11-05 955
    2017-11-06 860
    2017-11-07 900
    2017-11-08 875
    2017-11-09 902
    2017-11-10 901
    2017-11-11 943
    2017-11-12 947
    2017-11-13 907
    2017-11-14 895
    2017-11-15 910
    2017-11-16 914
    2017-11-17 839
    2017-11-18 930
    2017-11-19 943
    2017-11-20 859
    2017-11-21 942
    2017-11-22 913
    2017-11-23 898
    2017-11-24 891
    2017-11-25 912
    2017-11-26 906
    2017-11-27 871
    2017-11-28 922
    2017-11-29 924
    2017-11-30 938
    2017-12-01 929
    2017-12-02 996
    2017-12-03 976
    2017-12-04 903
    2017-12-05 879
    2017-12-06 922
    2017-12-07 903
    2017-12-08 925
    2017-12-09 926
    2017-12-10 904
    2017-12-11 855
    2017-12-12 906
    2017-12-13 882
    2017-12-14 879
    2017-12-15 850
    2017-12-16 876
    2017-12-17 904
    2017-12-18 877
    2017-12-19 902
    2017-12-20 880
    2017-12-21 818
    2017-12-22 855
    2017-12-23 806
    2017-12-24 843
    2017-12-25 790
    2017-12-26 840
    2017-12-27 855
    2017-12-28 929
    2017-12-29 901
    2017-12-30 874
    2017-12-31 890
    2018-01-01 879
    2018-01-02 885
    2018-01-03 890
    2018-01-04 873
    2018-01-05 877
    2018-01-06 880
    2018-01-07 909
    2018-01-08 881
    2018-01-09 899
    2018-01-10 932
    2018-01-11 954
    2018-01-12 886
    2018-01-13 923
    2018-01-14 915
    2018-01-15 944
    2018-01-16 945
    2018-01-17 938
    2018-01-18 912
    2018-01-19 901
    2018-01-20 904
    2018-01-21 940
    2018-01-22 854
    2018-01-23 886
    2018-01-24 934
    2018-01-25 909
    2018-01-26 916
    2018-01-27 886
    2018-01-28 975
    2018-01-29 907
    2018-01-30 919
    2018-01-31 931
    2018-02-01 959
    2018-02-02 981
    2018-02-03 984
    2018-02-04 1012
    2018-02-05 966
    2018-02-06 1029
    2018-02-07 987
    2018-02-08 802
    2018-02-09 149
    2018-02-10 967
    2018-02-11 1020
    2018-02-12 942
    2018-02-13 978
    2018-02-14 943
    2018-02-15 973
    2018-02-16 960
    2018-02-17 1032
    2018-02-18 981
    2018-02-19 959
    2018-02-20 960
    2018-02-21 1016
    2018-02-22 1043
    2018-02-23 1052
    2018-02-24 1041
    2018-02-25 975
    2018-02-26 929
    2018-02-27 986
    2018-02-28 1031
    2018-03-01 1036
    2018-03-02 1093
    2018-03-03 1120
    2018-03-04 1095
    2018-03-05 1083
    2018-03-06 1063
    2018-03-07 1081
    2018-03-08 1020
    2018-03-09 1033
     
  11. Jens_T

    Jens_T Avatar

    Messages:
    616
    Likes Received:
    1,396
    Trophy Points:
    93
    Location:
    Zurich
  12. Mykll

    Mykll Avatar

    Messages:
    674
    Likes Received:
    1,229
    Trophy Points:
    105
    Gender:
    Male
    Location:
    Davidian Bunker
    Not to mention on other sites devs have simply said "Go check out the Public API!" whenever someone questions numbers. But, seems like the API can't do that, huh?
     
    Jens_T likes this.
  13. Jens_T

    Jens_T Avatar

    Messages:
    616
    Likes Received:
    1,396
    Trophy Points:
    93
    Location:
    Zurich
    Perhaps this is a new functionality, @Chris @Undone can you help us here? It would be great if we had hard facts instead of speculation :)
     
  14. Undone

    Undone Trap Master SOTA Developer

    Messages:
    262
    Likes Received:
    662
    Trophy Points:
    28
    I believe @Chris meant increased activity. Due to the anonymity of the public api you can't get accurate concurrency from it. However, since I am the one crunching concurrency numbers I am happy tell you that they are indeed up =)
     
  15. MrBlight

    MrBlight Avatar

    Messages:
    2,388
    Likes Received:
    4,448
    Trophy Points:
    153
    Okay..
    So whats the give or take retention rate of new players past 2 months between persistance and now?
    What are the plans to increase this %? How can the community help?
    And would you give actual #s? And %s? Or any actual information?
     
    Mykll likes this.
  16. Lord Andernut

    Lord Andernut Bug Brigade - Bug Hunter

    Messages:
    3,258
    Likes Received:
    10,755
    Trophy Points:
    165
    Gender:
    Male
    Location:
    New Britannian Market
    I don't do the daily oracle quest, and many don't. So it works as an increased activity tracker but not of course for absolutes.
     
  17. kaeshiva

    kaeshiva Avatar

    Messages:
    1,875
    Likes Received:
    6,921
    Trophy Points:
    113
    Gender:
    Female
    I am online every day but often forget my oracle quest too..and my blessings....and to repair.....I'd leave town without my head if it were not attached.
    I wish there was an easier way to get this actual number of players...I don't understand why its something that seems to be intentionally 'hidden' ...kinda feel the same about 'xp to next level' though, and virtue values ....when stuff gets hidden from me it just makes me suspicious.
     
  18. Umuri

    Umuri Avatar

    Messages:
    529
    Likes Received:
    1,874
    Trophy Points:
    63
    Generic housecleaning in progress, speeding up some things that have hit performance issues when we had to move to new harddrives due to how big it was getting.
    But a new page courtesy of @Vladamir Begemot , the economy quick-page:
    Economy QuickPage
    Quickly and easily see quantity data across various time aggregates for various main resources.
     
  19. Vladamir Begemot

    Vladamir Begemot Avatar

    Messages:
    3,781
    Likes Received:
    7,214
    Trophy Points:
    153
    Gender:
    Male
    Thanks @Umuri, it's looking great and getting greater!
     
    Sean Silverfoot likes this.
  20. Umuri

    Umuri Avatar

    Messages:
    529
    Likes Received:
    1,874
    Trophy Points:
    63
    Due to the last few releases rendering me unable to play SOTA even in offline mode, I am declaring EOL for sotaSTATS, and making parts of the code available on github.
    https://github.com/umuri/SotaSTATS
    The poller/database populator at least is complete, so you can use that to start pulling your own copy of the data to play with.

    If individuals want the historical dump from SotaSTATS I will deal with those on a case by case basis as its hundreds of gb in size for the dataset to be transferred.
    Hope you all got some use out of it, and as always the code sucks. :)
     
    Titania Xylia likes this.