From renchap at cocoa-x.com Fri Jan 9 11:56:03 2009 From: renchap at cocoa-x.com (Renaud Chaput) Date: Fri, 9 Jan 2009 17:56:03 +0100 Subject: [Wowr-discuss] Hello world Message-ID: <93EF390D-33DC-4E70-BA34-72C262E14631@cocoa-x.com> Hi ! Seems it is the mailing list about wowr development, i hope we will improve wowr together. This project is very useful for me as i am developing a wow related website based on armory data. You can find an early beta version of my work here : http://beta.wowgossips.com/ Data is fetched once a day using wowr. As i will need to fetch many data (and many requests on the armory), i am planning to add some little things to wowr. First, the get_character call use 3 (2 with paul's github version) queries. I am storing character data in a db, and only update it when it changes. If characted has not changed (based on armory last update data), i dont need to fetch reputations, which uses one less request. I plan to split get_character into 2 methods (get_character_sheet and get_character_reputations), which will be called by get_character, but also allows you to fetch them alone. Are you ok with this ? I also plan to add an optional rate limiter on http queries (to avoid bans from armory if too many requests), i will try this later. I also need talents infos, and achievements/stats. If nobody adds it, i think i will try to do so. If you have comments about these features or ideas about implementation, feel free to reply. I forked Paul's github repos and will commit my changes here, if you want to have a look. renchap From renchap at cocoa-x.com Fri Jan 9 14:00:22 2009 From: renchap at cocoa-x.com (renchap) Date: Fri, 09 Jan 2009 20:00:22 +0100 Subject: [Wowr-discuss] Gemspec for wowr Message-ID: <49679EC6.2010309@cocoa-x.com> Hi, Is it a wowr.gemspec file for wowr ? I can not see it in Paul's GitHub repos, and i think it can be a good idea to add it, as it will allow users to install gem automagically built by GitHub : http://gems.github.com/, which allow them to easily install latest wowr version. renchap From peter+wowr at alastria.net Sun Jan 11 09:29:21 2009 From: peter+wowr at alastria.net (Peter Wood) Date: Sun, 11 Jan 2009 14:29:21 +0000 Subject: [Wowr-discuss] Gemspec for wowr In-Reply-To: <49679EC6.2010309@cocoa-x.com> References: <49679EC6.2010309@cocoa-x.com> Message-ID: <496A0241.3090206@alastria.net> Hey, > Is it a wowr.gemspec file for wowr ? No there isn't. I asked Ben about the gemspec for it when I took over as maintainer as I wanted to branch off 0.5.0. > I can not see it in Paul's GitHub repos, and i think it can be a good > idea to add it, as it will allow users to install gem automagically > built by GitHub : http://gems.github.com/, which allow them to easily > install latest wowr version. This is worth doing for sure! I'll have a go at creating a gemspec for it and we'll see how much github loves me. Peter. -- Peter Wood WowR Maintainer From peter+wowr at alastria.net Sun Jan 11 09:33:31 2009 From: peter+wowr at alastria.net (Peter Wood) Date: Sun, 11 Jan 2009 14:33:31 +0000 Subject: [Wowr-discuss] Hello world In-Reply-To: <93EF390D-33DC-4E70-BA34-72C262E14631@cocoa-x.com> References: <93EF390D-33DC-4E70-BA34-72C262E14631@cocoa-x.com> Message-ID: <496A033B.8070100@alastria.net> Hey Renaud, > Seems it is the mailing list about wowr development, i hope we will > improve wowr together. This project is very useful for me as i am > developing a wow related website based on armory data. Just to let you know I'm still trying to drag others onto the mailing list, for now it's just you and I. > First, the get_character call use 3 (2 with paul's github version) > queries. I am storing character data in a db, and only update it when it > changes. If characted has not changed (based on armory last update > data), i dont need to fetch reputations, which uses one less request. > I plan to split get_character into 2 methods (get_character_sheet and > get_character_reputations), which will be called by get_character, but > also allows you to fetch them alone. Are you ok with this ? I think you mean me, Peter, not Paul... although according to my parents I was almost a Paul. Yes this is one aspect of Wowr I would like to see changed, the problem is you can't break the API for backwards compatability. get_character will have to stay the same as it is now, and either by introducing two seperate stand alone functions or by flags being passed to get_character. Without looking at the source, I believe what you want should be relativly easy within the scope of the objects. > I also plan to add an optional rate limiter on http queries (to avoid > bans from armory if too many requests), i will try this later. Yes, most of us who have done any form of Armory pulling have been banned with 500 before now. I'm not strictly sure that this is an aspect for Wowr to solve however. In my own project, Syndicate, which is how I got onto using Wowr in the first place, we now have a background daemon that runs, analyses how much work it has to do in a block of time, and then calculates how long it needs to wait inbetween each poll. With a minimum wait period in between. In my experience you are okey to pull say 3-4 XML files simulataniously, providing you then wait a period of time afterwards. We are currently using a minimum wait time of 10 seconds, and on average with our work load it's waiting about 180 seconds between each character pull. I've spoken to the guy who runs be.imba.hu, and his back ends wait 2 seconds and get away with it. Although he indicated he had different gatherers about the internet to distribute the load. It may be that you want to write a small queuing library that uses Wowr as a seperate gem? > I also need talents infos, and achievements/stats. If nobody adds it, i > think i will try to do so. Achievements is my next priority to develope into Wowr, these and statistics go hand in hand and need to be developed in a none static manner. For example, we can't have statistics objects with hundreds of predefined variables. If we do that we will be playing catch up with Blizzard forever. I'm excited to see a different view of Wowr then my own. I'd like the option for data to be able to be pulled in a more selective manner. My personal long term goal for Wow is that the objects that are returned can be used to requery via functions, rather then having to recostruct the call. For example, in your case, you do a get_character_sheet and you get their basic information, but they you can call character.get_factions. Not final API wording, but I hope you understand what I suggest. Peter. -- Peter Wood WowR Maintainer From peter+wowr at alastria.net Sun Jan 11 09:22:17 2009 From: peter+wowr at alastria.net (Peter Wood) Date: Sun, 11 Jan 2009 14:22:17 +0000 Subject: [Wowr-discuss] Hello world In-Reply-To: <93EF390D-33DC-4E70-BA34-72C262E14631@cocoa-x.com> References: <93EF390D-33DC-4E70-BA34-72C262E14631@cocoa-x.com> Message-ID: <496A0099.3070200@alastria.net> Hey Renaud, > Seems it is the mailing list about wowr development, i hope we will > improve wowr together. This project is very useful for me as i am > developing a wow related website based on armory data. Just to let you know I'm still trying to drag others onto the mailing list, for now it's just you and I. > First, the get_character call use 3 (2 with paul's github version) > queries. I am storing character data in a db, and only update it when it > changes. If characted has not changed (based on armory last update > data), i dont need to fetch reputations, which uses one less request. > I plan to split get_character into 2 methods (get_character_sheet and > get_character_reputations), which will be called by get_character, but > also allows you to fetch them alone. Are you ok with this ? I think you mean me, Peter, not Paul... although according to my parents I was almost a Paul. Yes this is one aspect of Wowr I would like to see changed, the problem is you can't break the API for backwards compatability. get_character will have to stay the same as it is now, and either by introducing two seperate stand alone functions or by flags being passed to get_character. Without looking at the source, I believe what you want should be relativly easy within the scope of the objects. > I also plan to add an optional rate limiter on http queries (to avoid > bans from armory if too many requests), i will try this later. Yes, most of us who have done any form of Armory pulling have been banned with 500 before now. I'm not strictly sure that this is an aspect for Wowr to solve however. In my own project, Syndicate, which is how I got onto using Wowr in the first place, we now have a background daemon that runs, analyses how much work it has to do in a block of time, and then calculates how long it needs to wait inbetween each poll. With a minimum wait period in between. In my experience you are okey to pull say 3-4 XML files simulataniously, providing you then wait a period of time afterwards. We are currently using a minimum wait time of 10 seconds, and on average with our work load it's waiting about 180 seconds between each character pull. I've spoken to the guy who runs be.imba.hu, and his back ends wait 2 seconds and get away with it. Although he indicated he had different gatherers about the internet to distribute the load. It may be that you want to write a small queuing library that uses Wowr as a seperate gem? > I also need talents infos, and achievements/stats. If nobody adds it, i > think i will try to do so. Achievements is my next priority to develope into Wowr, these and statistics go hand in hand and need to be developed in a none static manner. For example, we can't have statistics objects with hundreds of predefined variables. If we do that we will be playing catch up with Blizzard forever. I'm excited to see a different view of Wowr then my own. I'd like the option for data to be able to be pulled in a more selective manner. My personal long term goal for Wow is that the objects that are returned can be used to requery via functions, rather then having to recostruct the call. For example, in your case, you do a get_character_sheet and you get their basic information, but they you can call character.get_factions. Not final API wording, but I hope you understand what I suggest. Peter. -- Peter Wood WowR Maintainer From renchap at cocoa-x.com Sun Jan 11 11:07:30 2009 From: renchap at cocoa-x.com (renchap) Date: Sun, 11 Jan 2009 17:07:30 +0100 Subject: [Wowr-discuss] Hello world In-Reply-To: <496A0099.3070200@alastria.net> References: <93EF390D-33DC-4E70-BA34-72C262E14631@cocoa-x.com> <496A0099.3070200@alastria.net> Message-ID: <496A1942.2000500@cocoa-x.com> Peter Wood a ?crit : > Hey Renaud, Hey, > >> Seems it is the mailing list about wowr development, i hope we will >> improve wowr together. This project is very useful for me as i am >> developing a wow related website based on armory data. > > Just to let you know I'm still trying to drag others onto the mailing > list, for now it's just you and I. Ok. I hope more people will come to help us :) >> First, the get_character call use 3 (2 with paul's github version) >> queries. I am storing character data in a db, and only update it when >> it changes. If characted has not changed (based on armory last update >> data), i dont need to fetch reputations, which uses one less request. >> I plan to split get_character into 2 methods (get_character_sheet and >> get_character_reputations), which will be called by get_character, but >> also allows you to fetch them alone. Are you ok with this ? > > I think you mean me, Peter, not Paul... although according to my parents > I was almost a Paul. Sorry, i know a Paul Wood ;) Yes this is one aspect of Wowr I would like to see > changed, the problem is you can't break the API for backwards > compatability. I plan to maintain backward compatibility, but i will add some methods/classes to handle this. > get_character will have to stay the same as it is now, and either by > introducing two seperate stand alone functions or by flags being passed > to get_character. Without looking at the source, I believe what you want > should be relativly easy within the scope of the objects. I started to work on this : http://github.com/renchap/wowr/commit/11e999930ebf4561772858000cd1edc0a1cf3673 I am now working on a InfosCharacter.to_fullCharacter method which will allow you to convert an InfosCharacter to a FullCharacter by copying attributes and fetching reputations.xml. I will also add an API method to get an InfosCharacter. Maybe i can add an optional method to get_character to select which informations you want. What do you prefer ? >> I also plan to add an optional rate limiter on http queries (to avoid >> bans from armory if too many requests), i will try this later. > > Yes, most of us who have done any form of Armory pulling have been > banned with 500 before now. I'm not strictly sure that this is an aspect > for Wowr to solve however. Wowr should at least give information about how many HTTP requests have been done so far, with a way to reset this counter. This counter should also be armory specific (i think wowarmory.com and eu.wowarmory.com dont share the same limit counter). > In my own project, Syndicate, which is how I got onto using Wowr in the > first place, we now have a background daemon that runs, analyses how > much work it has to do in a block of time, and then calculates how long > it needs to wait inbetween each poll. With a minimum wait period in > between. > > In my experience you are okey to pull say 3-4 XML files simulataniously, > providing you then wait a period of time afterwards. We are currently > using a minimum wait time of 10 seconds, and on average with our work > load it's waiting about 180 seconds between each character pull. > > I've spoken to the guy who runs be.imba.hu, and his back ends wait 2 > seconds and get away with it. Although he indicated he had different > gatherers about the internet to distribute the load. > > It may be that you want to write a small queuing library that uses Wowr > as a seperate gem Maybe we can use a plugins system for HTTP queries, and provide an example plugin which implement a simple rate limiter ? I dont see how a separate gem can provide this kind of functionnality. If you have more informations, I am interested :) Or we can just leave this to the developer which uses Wowr by providing counters, as said above. > >> I also need talents infos, and achievements/stats. If nobody adds it, >> i think i will try to do so. > > Achievements is my next priority to develope into Wowr, these and > statistics go hand in hand and need to be developed in a none static > manner. For example, we can't have statistics objects with hundreds of > predefined variables. If we do that we will be playing catch up with > Blizzard forever. I think we will need some kind of database (plain yaml files should be ok) to store all achievements locally, with a built-in rake task to create this database from armory and update it. I need achievements to store any changes in a database. In the achievements main xml, you get how many achievements points in each categories. I personnaly need to be able to request categories details if the category total have changed. > I'm excited to see a different view of Wowr then my own. I'd like the > option for data to be able to be pulled in a more selective manner. > > My personal long term goal for Wow is that the objects that are returned > can be used to requery via functions, rather then having to recostruct > the call. > > For example, in your case, you do a get_character_sheet and you get > their basic information, but they you can call character.get_factions. > Not final API wording, but I hope you understand what I suggest. I agree on this. On an ideal world, i think all request should only be made on-demand. Like (pseudo-code) : char = Char.new(:name => 'Mychar') # (no requests made) char.level => 80 # a request have been done to fetch level, and others informations provided by the needed xml char.reputations => Hash... # Another request for reputations We can also provide a manual way to speficy which informations should be loaded, like giving an array to constructor, or methods to load these informations (and use an algorythm to minimize number of requests needed for a specific st or informations) > > Peter.