Translation
Hi spiludviklers (figured that would be the correct term).
Does any of you have good experiences with outsourcing QA to professional QA companies? And if yes could I get you to share those companies, I guess it could be beneficial not just for me (us) but for other people considering outsourcing that part of their work.
Cheers,
/E
Good experiences with outsourced QA
Re: Good experiences with outsourced QA
As a QA/UX Manager myself, I can only offer an opinion from the other side of the fence. What I can say is thate outsourcing QA to another company is a very viable solution, as it adds objectivity and a new look on things (which is always a good thing when it comes to QAing parts that are meant to be secure, I.E. account/login information etc.).
That being said, the main factor is objectivity - you could hire a professional QA firm to do the work for you, or you could hire yuor own QA person who can do it for you, as long as they're objective (Tests the game and reports their findings, and otherwise aren't a direct part of the development of the game), both can be used. Just know that a QA company will most likely have many clients, whereas a QA Manager working for you directly only have one - you. In short, the latter can contribute 100% of his time on your game (and doing related research as well as finding tester), whereas a QA company will have multiple clients to juggle with, though usually they also have the manpower (and resources) to do so without suffering much in the quality of their work.
That said, I cannot really povide any good answers until I know more specificly what you mean by "QA" (The term have been used wildly and so, I always makes it a point to know what someone specificly means when saying "QA"). If it includes UX (User Experience), then both are good solutions, though having your own QA means he/she may better know what your goal with the game is, and thus more efficiently design tests. The negative point of having your own QA is that he may get "used" to testing the same game over and over, and thus suffer a drop in objectivity - which is why having a QA who does UX is essential (in my book atleast), as he continually works with the objective findings brought through external-user/target user tests.
That being said, the main factor is objectivity - you could hire a professional QA firm to do the work for you, or you could hire yuor own QA person who can do it for you, as long as they're objective (Tests the game and reports their findings, and otherwise aren't a direct part of the development of the game), both can be used. Just know that a QA company will most likely have many clients, whereas a QA Manager working for you directly only have one - you. In short, the latter can contribute 100% of his time on your game (and doing related research as well as finding tester), whereas a QA company will have multiple clients to juggle with, though usually they also have the manpower (and resources) to do so without suffering much in the quality of their work.
That said, I cannot really povide any good answers until I know more specificly what you mean by "QA" (The term have been used wildly and so, I always makes it a point to know what someone specificly means when saying "QA"). If it includes UX (User Experience), then both are good solutions, though having your own QA means he/she may better know what your goal with the game is, and thus more efficiently design tests. The negative point of having your own QA is that he may get "used" to testing the same game over and over, and thus suffer a drop in objectivity - which is why having a QA who does UX is essential (in my book atleast), as he continually works with the objective findings brought through external-user/target user tests.
Re: Good experiences with outsourced QA
A few years back I started Playdeads QA efforts. That has resulted in a full time QA guy and a part time guy who helps out.
Back when I started I would have said that that was enough. But then I went through my first certification process! That was X360, which was actually peace of cake compared to the later PS3 certification(and PS Vita which I did almost alone)...
We found at Playdead, that if we concentrate on making sure the game runs well and bug free, then we use 3rd party QA to verify all the stuff we don't like to spend resioucres getting good at. Being "certification requirements". So what we do is that we keep the game rather bug free all the way until we're close to a serious release candidate and than we do a first pass with external QA. Who'll return to us with a lot of platform specific bugs and of course some more "grind" based bugs from the game.
This means that we now have a solid list of eveything that the game needs to "be able" to do to pass certification. At this without spending so much time reading up on it.
Then we do 2-4 extra passes with external QA.
So yeah we've had good experience with using external QA(https://www.vmc.com/games/), but be ware that they really wanna sell you a huge package. So a good idea here is to be super critical on what actually is needed. When we first approached VMC they wanted to throw 5000 test cases at LIMBO. And I was like WHAT!!?
Then you find out that 2000 test cases if for online multiplayer, and 1000 is for dance-mat support(joking)... And then you end up with a nice set of tests that does the thing.
I can't see how any "decent" size developers can ship a game without help from external QA. But then again there's a lot of buggy game out there, and they have to come from somewhere...
- Peter
Back when I started I would have said that that was enough. But then I went through my first certification process! That was X360, which was actually peace of cake compared to the later PS3 certification(and PS Vita which I did almost alone)...
We found at Playdead, that if we concentrate on making sure the game runs well and bug free, then we use 3rd party QA to verify all the stuff we don't like to spend resioucres getting good at. Being "certification requirements". So what we do is that we keep the game rather bug free all the way until we're close to a serious release candidate and than we do a first pass with external QA. Who'll return to us with a lot of platform specific bugs and of course some more "grind" based bugs from the game.
This means that we now have a solid list of eveything that the game needs to "be able" to do to pass certification. At this without spending so much time reading up on it.
Then we do 2-4 extra passes with external QA.
So yeah we've had good experience with using external QA(https://www.vmc.com/games/), but be ware that they really wanna sell you a huge package. So a good idea here is to be super critical on what actually is needed. When we first approached VMC they wanted to throw 5000 test cases at LIMBO. And I was like WHAT!!?
Then you find out that 2000 test cases if for online multiplayer, and 1000 is for dance-mat support(joking)... And then you end up with a nice set of tests that does the thing.
I can't see how any "decent" size developers can ship a game without help from external QA. But then again there's a lot of buggy game out there, and they have to come from somewhere...
- Peter
Re: Good experiences with outsourced QA
Thank you for the input.
@Bo: I am mostly looking towards technical QA, traditional bug finding and testing on different platforms, configurations etc. I intend to keep user experience testing in house for now.
@Buchardt: Thanks for the link to vmc
Keep more links coming if you have good experiences (or some experiences) to share.
@Bo: I am mostly looking towards technical QA, traditional bug finding and testing on different platforms, configurations etc. I intend to keep user experience testing in house for now.
@Buchardt: Thanks for the link to vmc
Keep more links coming if you have good experiences (or some experiences) to share.
Re: Good experiences with outsourced QA
Can definitely recommend Testology in UK, which have helped us keeping our games "almost" bug free for quite some years now. Good guys.EKnudsen wrote:Thank you for the input.
@Bo: I am mostly looking towards technical QA, traditional bug finding and testing on different platforms, configurations etc. I intend to keep user experience testing in house for now.
@Buchardt: Thanks for the link to vmc
Keep more links coming if you have good experiences (or some experiences) to share.
Jeppe Bisbjerg
Director of Publishing
Director of Publishing
Re: Good experiences with outsourced QA
Awesome, thanks
Re: Good experiences with outsourced QA
Yes, I have some experience with such company as NixSolutions. You can easily get more imformation about their QA service here http://www.nixsolutions.com/services/qu ... -services/. They are professionals and I can recommend them.
Re: Good experiences with outsourced QA
Outsourcing it is very interesting. Many big companies use outsourcing in their business. But there is also such thing as outstaffing https://qubit-labs.com/outsourcing-vs-outstaffing/ It is not the same, but it is also a good solution.
-
BubbleWrapBoy
- Level 8 - Summoner
- Posts: 81
- Joined: 26 Nov 2010, 12:44
Re: Good experiences with outsourced QA
Man... This, right here, is the reason people like Trump gets elected and Britain leaving EU. On their web it even says 'outstaffing provides the ability to hire great professionals without having to pay high price and taxes;'. Disgusting.thowans76 wrote:Outsourcing it is very interesting. Many big companies use outsourcing in their business. But there is also such thing as outstaffing https://qubit-labs.com/outsourcing-vs-outstaffing/ It is not the same, but it is also a good solution.
Re: Good experiences with outsourced QA
The best solution now is to create an offshore development center in Eastern Europe. It has helped my company a lot to cut costs.BubbleWrapBoy wrote: ↑28 Jan 2017, 03:39Man... This, right here, is the reason people like Trump gets elected and Britain leaving EU. On their web it even says 'outstaffing provides the ability to hire great professionals without having to pay high price and taxes;'. Disgusting.thowans76 wrote:Outsourcing it is very interesting. Many big companies use outsourcing in their business. But there is also such thing as outstaffing https://qubit-labs.com/outsourcing-vs-outstaffing/ It is not the same, but it is also a good solution.