Broad based BETA test of 921 software

[BCH]

Scott, this may be a suggestion you can offer to your contacts at Dish.

Since the number of 921 receivers in consumers hands are rather limited why not open the distribution of the latest beta software to all that volunteer to accept it. It would provide a broader base for Dish to ascertain the effectiveness of the fixes before the flood gates are opened for 921 distributions to the public.

 

[JosephF]

What do you do if the Beta software is worse?

While E* can target certain populations of receivers (as we know from their gradual rollouts of new versions) they can still only target that population with one version of software. Thus, unless they reverted their stream back to an old version, you would have no way to opt out after the fact.

I agree it would be a good idea to do this, but don't see how it could be done without a major revamping of how E* streams downloads.

 

[Scott Greczkowski]

Yup knowing how Dish does the beta downloads this would be VERY hard for them to do.

Remember that this software is written in ENGLAND and is tested using fake satellite signal in a lab, while it may work good in the Lab they never know how it works until they test it on real satellite signals here in the US.

Of course here in the US under real conditions there could be a lot of bugs that were not seen in the lab conditions.

 

[BCH]

I acknowledge that I am not aware of how Dish targets which receivers get beta software. It would be my assumption that receiver IDs are used to determine which receivers will get software updates both final and beta.

What would I do if the beta release was worse? I would probably take the same actions as current beta testers. Report my observations to Dish. That is why i suggested that only volunteers would receive this update. In my particulay case I have a spare 6000 and a spare 811 which can be used if the 921 is rendered totally unusable.

Having done beta testing for other companies I am also aware that the problem of a NDA may exist. A NDA is really only needed if the software contains new or innovative methods that may provide a competitor with some type of advantage. If the update only repairs known items which are not functioning properly it is not likely that a NDA will be needed.

 

[Scott Greczkowski]

I do wish Dish Network had some kind of Beta Tester Application so users who want to test could apply to become a tester.

There are a lot of us out here that are good at breaking new software when its released, wouldnt it be great for us to break beta software so they could fix it before its released to the general public?

 

[TheForce]

Laboratory testing never seems to replicate real world conditions. This is why there is a need for beta testing. The problem with opening up the beta test to an uncontrolled group who are, by chance, first to receive product is that you have on your beta test team people who have no clue on how to conduct testing in a logical and scientific fashion. I would venture to say that NONE of the current people, including Mark and Scott who have received 921's have ever been through a curriculum of formal education in scientific data collection, environmental stress testing, or similar program. The reason for this type of training is so that problems discovered can be reduced to specifics with the domain and range of the issue defined before reporting to the next level of test team management. beta testing is not just playing around with the device and then completing a report of your wish lists. It is a thorough process of discovery of the issues and then defining each issue as to as many ways to generate the problem as you can and building an instruction set so that the problems can be first replicated among all beta testers and then the test team management.
It is for this reason that the E* beta testers are selected from employees in the technical division as these employees can be controlled and instructed on the procedures they will follow.
I recall a beta team that was assembled for a company that makes broadcast TV test equipment. I was offered to be on the team. When the first set of instructions were sent out for testers to review, over 80% of the volunteers resigned. They did not have the time nor desire to adhere to the process of scientific data collection.
I favor keeping the E* beta testing in the hands of trained techincians who have an understanding of the equipment and who can be controlled by E*

I think if you recall, the OTA bug was announced before the first public release of the 921 and the fact that the 921's began shipping before this bug was fixed had to do with marketing, not technical. If you left the decision to release a device after ALL known problems are fixed the 921 would never be out the door. The decision to release with this bug still present was probably based on an estimation of when the bug fix in an updated version of the software would be ready. It was my understanding from reports here that the reason for the delay in this new bug fix update had to do with issues related to uplink and not the bug itself.

The check switch bug, however was indeed discovered by a Dish Net dealer who sold a 921 he opened for himself and then decided to sell it. Since the new owner had a different dish setup the check switch bug surfaced and failed. Issues like this will surface from time to time and will be reported by end users in the field. These are called corner cases and while an all volunteer beta team would probably discover these the time to uncover 99.9% of all corner cases and effect a fix would be far more lengthy. I know of two such beta tests ongoing now that I am involved with and both of these beta programs have been ongoing for 2 years on one and 5 years on the other. They are buy-in beta programs so the company is making money selling beta product.
The OTA bug is a global one discovered by the curreent beta test team . The Check switch bug is a corner case bug that only surfaces when certain user conditions are present such as purchasing a used 921 or one that was previously activated for use and them moved to a new dish antenna system.

I believe these are the only two bugs currently known and both are said to be fixed in the new software when released.

 

[c150flyboy]

Remember that this software is written in ENGLAND

Why is it written in England? Is it cheaper to write software there? :?:

 

[JosephF]

Because E* has three different Engineering groups. The main one in Englewood (ETC), on in Atlanta (EDN) and one in the UK (Eldon).

 

[markdl]

Laboratory testing never seems to replicate real world conditions. This is why there is a need for beta testing. The problem with opening up the beta test to an uncontrolled group who are, by chance, first to receive product is that you have on your beta test team people who have no clue on how to conduct testing in a logical and scientific fashion. I would venture to say that NONE of the current people, including Mark and Scott who have received 921's have ever been through a curriculum of formal education in scientific data collection, environmental stress testing, or similar program.

Actually, Don, you'd be incorrect in my case. I have been through a program like you describe. The first couple of weeks I had my 921 I spent much more time writing reports about it than I did watching it (on the order of 10 to 1)! I can definitely say from personal experience that beta testing takes a huge amount of work when done properly, and that most people who say they want to be beta testers would never consider donating that much time and effort.

I think if you recall, the OTA bug was announced before the first public release of the 921 and the fact that the 921's began shipping before this bug was fixed had to do with marketing, not technical. If you left the decision to release a device after ALL known problems are fixed the 921 would never be out the door. The decision to release with this bug still present was probably based on an estimation of when the bug fix in an updated version of the software would be ready. It was my understanding from reports here that the reason for the delay in this new bug fix update had to do with issues related to uplink and not the bug itself.

That's correct. At the time I felt it was a disservice to users who were going to buy the 921 without knowing about it's biggest problem.

The OTA bug is a global one discovered by the curreent beta test team . The Check switch bug is a corner case bug that only surfaces when certain user conditions are present such as purchasing a used 921 or one that was previously activated for use and them moved to a new dish antenna system. I believe these are the only two bugs currently known and both are said to be fixed in the new software when released.

There are many more bugs identified with the 921, and the next software version fixes a lot of them. Not all, but quite a few (23 by my count).

 

[Mark_AR]

I've been a BETA tester for several companies all the way back to Ziff Davis (Now ZDNET, the TechTV people) when they launched their own online service back in 96.

The reports are the worst part. But I think its kewl to try and crash things by doing stuff that the normal user may never think of doing.

The hardest part is to DUPLICATE crashes. Without duplication, it is hard to track down where the failure was.

At least if there was a logging function that recorded maybe the last 5 minutes of activity (similar to the DVR view buffer) it would make debugging simpler for the techies.

Just my 2¢ worth.

 

[Scott Greczkowski]

The PVR receivers with Beta Software DO have logging software and can upload to the Echostar techs.

Note this logging software is NOT in the release software.

 

[SlicerMDM]

I do wish Dish Network had some kind of Beta Tester Application so users who want to test could apply to become a tester.

There are a lot of us out here that are good at breaking new software when its released, wouldnt it be great for us to break beta software so they could fix it before its released to the general public?

Now THAT'S an innovative idea! Actually test (and fix) the software before releasing it! Someone should really tell Charlie about this new concept.

 

[TheForce]

"Actually, Don, you'd be incorrect in my case. I have been through a program like you describe. "

Wasn't sure but I do give you credit for doing a bang up job in your first public report on the 921. You 'da man when it comes to consumer test reports, that's for sure!

I did a program that took two years in time, part time through the Florida Sea Grant and University of Florida. Nothing to do with electronics but strictly in methods and techniques in designing scientific data collection methods to include basic crime scene investigation. The principles are generic enough to apply to any technology.
In much of my past work I found the report writing was the most interesting as it often would surface missing data in the tests, sending you back to the hardware or software to redo and redo until all the permutations were tested.

If one has fun breaking stuff, that is usually easy and not really beta testing, the difficult part is to describe in a formal report all the ways that device can be broken by following a procedure and then keeping your mouth shut under the NDA. One beta NDA I did you weren't even permitted to say you were a beta tester and you had to be selected, if you applied it automatically disqualified you. That was Auto Desk. I don't know how they operate today but that was 10 years ago.

 

[MikeD-C05]

I thought all customers of Dishnetwork were Beta testers with new receivers.

Don't we post all the bugs here on this forum and others. A cheap way to have Dish get the information it needs to fix the software bugs. Not very proffessional but I am sure much more affordable for cheap old Charlie.

 

[Scott Greczkowski]

Mike your 100% correct about that.

As you may know Dish Network asked us and DBSTalk to run "Official" 921 support boards.

The only thing really official about them is it officially makes it easier for the 921 guys in England to follow threads about the 921's.

Nothing more official then that. You will never see an actual Dish Network post at eaither forum. They are not allowed to.

Basically we are cheap tech support for Dish, but hey I don't care, I run the site to help people so I went with it.

 

[kvhollis]

Mike your 100% correct about that.

As you may know Dish Network asked us and DBSTalk to run "Official" 921 support boards.

Nothing more official then that. You will never see an actual Dish Network post at eaither forum. They are not allowed to.

Thats too bad Dish doesn't post messages. If you ask you to host a "forum" they should participate as well as help with the bandwidth.

I'd personally like to see their participation in other forums as well as be able to see their point of view.

Khan

 

[boy921]

Yup knowing how Dish does the beta downloads this would be VERY hard for them to do.

Remember that this software is written in ENGLAND and is tested using fake satellite signal in a lab, while it may work good in the Lab they never know how it works until they test it on real satellite signals here in the US.

Of course here in the US under real conditions there could be a lot of bugs that were not seen in the lab conditions.

Scott, couldn't the Brits come to the Yanks and remain in a real lab until they resolve the issues? It seems that it would be a worth while expense for Dish Net. It sounds like lab rat testing.

 

[SimpleSimon]

Actually, most (all?) of what's posted i this thread is old news - already hashed out over in the DBStalk forum.

Eldon DOES have people over here, and apparently Mark Lamutt meets with them often.