APL97 Conference Survey Results by Dick Holt This article summarizes the APL97 attender feedback survey. It compares results, where possible, with similar prior surveys. It describes attenders satisfaction with various features of APL97. The purpose of the article is to help the APL/J community to improve future conferences. Briefly, attenders rated APL97 as a huge success. 10 of 14 Con- ference features rated higher than in two comparable prior surveys, most of them notably so. The 4 exceptions (below), may appear to be comparable to prior surveys, but for varying reasons, are not. (1) "Proceedings" rated slightly lower at APL97 than in the 1994 survey (Proceedings were unavailable at APL96). Some APL97 attenders noted that Proceedings on a CD ROM and weren't easily readable on-site (but only 30% looked at the CD ROM -- see Q17). Even so, the CD ROM helped APL97 to reduce lead times and to keep costs low. The APL97 Proceedings rating reflects its format, not its content. As a result, it's not comparable to prior survey data. (2) In contrast, APL97 attenders rated the Software Library highly compared to prior surveys. Again, this reflects the CD ROM format rather than content. (3 & 4) "Cost of conference" and "Other costs" aren't comparable to prior surveys. We changed the APL97 questionnaire to distinguish, for the first time, between "Cost of conference" (e.g. registration and banquet), and "Other costs" (e.g. travel and lost earnings). "Cost of conference" at APL97 rated in 1st place, clearly showing APL97's success in reducing those costs over which its organizers had control. Yet, "other Costs" rated low. In statistical jargon, APL97 data for "Cost" and "Other Costs" are confounded if compared with prior survey data. Now to the Survey Questions and Answers. Q1 asked attenders to rate APL97 features on a scale of 1 to 5 (most-to-least satisfied). Average ratings, listed in order of decreasing satisfaction, are shown below: Table 1. Satisfaction with APL97 Conference Features * = ratings higher than 94/96 surveys n = not asked in 94/96 surveys x = not comparable to 94/96 surveys cost of conference 1.10 x social/vacation 1.77 * location 1.29 * papers 1.81 * website 1.32 n proceedings 1.84 x program handbook 1.36 n other costs 1.89 x archives, etc. 1.43 n dinner cruise 2.03 * tutorials 1.58 * business opportunities 2.06 * software library 1.59 x job information 2.19 * accommodation 1.68 * vendor hospitality 2.34 n workshops 1.72 * posters, displays 2.58 * vendor forums 1.74 * others? -- only 1 response Small differences in ratings (+/- 0.2) probably aren't significant. See References 1 and 2 for 94/96 survey data. Q2: What was the most useful feature, and why? Multiple responses are combined below: - Arrangements: Very good arrangements; good programs; program was well thought out - Workshops: learned about very pertinent stuff; got to try things; best way to learn is to try; participants received real hands-on experience and training in advanced features. - Tutorials: practical how to do; excellent content; stream of J tutorials; subjects were mostly practicable; APL info. - Workshop and vendor forums - Convenient (for us in Toronto). Very knowledgeable crowd. - Vendor forums: clear understanding of current market; got the latest developments - Meeting people - CD rom proceedings -- may make software accessible, pretty current Q3: What was the least useful feature, and why? - Vendor forums too far from presentations (multiple comments) - Workshops -- great idea but poor presentation. Overhead info too dead, too small and too dim to be read or used. - Workshops -- topic selection poor for me - CD rom proceedings -- can't read papers at conference or on trolley; Missing proceedings (see Q17); hosp suites instead of open is best. Q4 and Q5 concern Conference attendance. Overall, 79% of respon- dents had attended a prior SIGAPL conference, and 84% said that they would attend a future one (90% and 92% at APL96). When asked why they might attend a future SIGAPL Conference, respondents said: "I use APL and J professionally; Fun and fruitful; It's my business; It has become a habit; Interested in APL employment and meeting other APLers; Greater APL involvement; Useful info; Worth attending; Business; Helps to keep abreast of developments; For apl related information, products, contacts; Meeting people; Stimulating to get new ideas." Q6: Are you a member of SIGAPL? Y=70% (APL96: Y=90%) Q7: Years of experience with APL? With J? A simple answer: an average of 13.6 years with APL, and 2.7 years with J. Prior surveys showed a trend of increasing years of APL experience (15.6 in 94, 16.9 in 96). That trend is lightly reversed at APL97. A more interesting answer: about 28% of respondents are experienced in both APL and J. Their joint years of APL and J experience are shown in the 2x12 table below: APL yrs: 6 20 8 15 14 21 25 8 13 20 20 25 J yrs: 2 6 1 5 1 5 3 2 1 1 6 1 Similar cross-tabulations, mainly of interest to vendors, may be constructed using the APL97SRV.ZIP file mentioned below. Q8: Your State/Province/Country? This question is key to the analytic validity of this article. These analyses are valid only if the questionnaire respondents are a statistically representative of the population of all APL97 attenders. A 10,000 draw bootstrap (Monte Carlo) shows that the survey sample is indeed representative of APL97 attenders. # of APL97 # of Survey Attenders Responses Bootstrap Bootstrap from: from: mean 2 sigma Canada 96 11 16.5 5.8 USA 82 18 14.1 5.6 Europe 61 12 10.5 5.0 Other 11 2 1.9 2.4 Total 250 43 See Reference 2 for details of the bootstrap technique. The US is somewhat over-represented, and Canada is somewhat under-represented in the sample, however both are within 2 sigma limits of their bootstrap means. "Europe" and "Other" are easily on the mark. Attenders came from 4 Canadian Provinces (mostly Ontario), 20 US states, 10 European nations (including Russia), and from Japan, Saudi Arabia, and South Africa. The survey response rate was 17%, well below the 27% response rate for the APL96 survey. Why? Because APL97 survey forms were available only adventitiously, rather than being inserted in Conference Bags at registration. Q9: "Your occupation?" showed a wide array of responses. Most were in the area of software development (32%) and education (16%), both very close to APL96, followed by actuary (7%) and scientist (7%). Others include student, engineer, financial analyst, webmaster, consultant, business, and others. Q10: "In what applications have you used APL/J?" Applications mentioned most often were: banking/finance .39 insurance/actuary .09 education .30 engineering .07 business/manufacturing .12 Many respondents named multiple applications including advertising, medicine, oil, law, elections, beverage, forestry, travel, biology, nutrition, and others. Q11. Are you actively developing code in APL, J, or other Array languages? APL only : 56% J only : 7% Both APL and J : 9% Other array also: 14% Q12: Where did you first hear about APL97?, and Q13: What was your major source of ongoing information about APL97? mailing email c.l.a. Vector QQ Nouvelles Other Q12: 14% 26% 19% 2% 5% 0% 23% Q13: 16% 53% 21% 7% 7% 5% 14% APL97 publicity reached far and wide. Web sites were most often cited as "other." Electronic publicity was the most wide reaching. Q14 and Q15 concerned Vendors. 56% of survey respondents visited vendor Hospitality Suites, and of those who visited, 86% felt that they provided a useful contribution to APL97. Q16 asked for comments on Exhibitors/Vendors. Opinions were mixed, and physical location was apparently an issue: On the plus side: "Hospitality was one of the main reasons for attending, finding out about new products and watching them demonstrated; Suites harder to get to but receive more attention there; More emphasis on the seriously interested; Hospitality is better; Hospitality is better more focused." On the minus side: "I prefer ... a single exhibitor area. This was too scattered; Exhibits were close, hospitality was remote; Prefer central open exhibit area, really *near* lecture rooms; Too far from meeting room; Hospitality suites too formal ... a barrier to go there; Wish vendor sites near coffee between papers; Hospitality suites should have been in same building." One respondent noted that the survey didn't distinguish among vendors, and that some vendors had better hospitality suites than others. Almost all one-word responses to Q16 favored the "open- concept" over the hospitality suite approach for Vendors' exhibits. Q17: CD ROM: Only 30% of respondents looked at the CD ROM, and 60% of those who did found it useful. But this isn't the full story. Many noted that they couldn't easily read the CD at the Conference. Below is an example of suggestions for future CD ROMS. See also Q3 (least useful feature). " 1) The CD was a good idea but it went too far: Please distribute attendance list ON PAPER at start of Conf (later a more updated version on www is fine too). 2) I do like to have the proceedings ON PAPER. If not, my boss will not consider any contribution I make to the Conf ... as valuable and will not pay for my trip. 3) I do like to have the proceedings on paper to read on the first night of the conf. 4) I appreciate a low cost conf, but am willing to pay more for proceedings on paper." You may order the APL97 CD ROM from www.torontoapl.org for $C20, postage paid world-wide, via cheque, money order, Visa, Master, or Amex. Toronto SIGAPL will make the CD ROM electronically available and updated at, or linked to, its web page (www.torontoapl.org). Q18: Was the Program Handbook useful? Y = 74% Program handbook comments: "Add good map of the area ... is needed; Pg 18-20: begin-end times, like in the flyer, easier to read than the begin time only; Set start times on print-out more carefully; Suggest adding start and end times; Keep schedule in center -- easy to open and find; Put day, time, and location beside each description to change more bigger one size; Perhaps include a floor map of room locations; It was exactly up to its task." Q19a: Want Annual APL/J conference in Toronto? Y=28% N=28% B=44% Q19b: Want a Bi-annual conference? Y=49% N=19% B=37% Note: some respondents may have interpreted the word "bi-annual" (twice/yr) in Q19b to mean "biennial" (every 2 years). Q20a: Are you aware of the APL Skills Database? Y=62% Q20b: Other services wanted? There were a lot of blanks in response to Q20b. Items mentioned were: "Part-time employment; more web stuff; Re'sume' marketing service?; internet conference; 1) debugging tools, 2) Ecklers (Paul Davidson) coding protocols, 3) version control software; and workshop, - contest Q21: Topics for future conferences?: Verbatim responses were: "Performance consideration in real-life software developed in APL/J; Application in scientific & mathematical problems; OLE controls (i.e. getting them to work in APL); General tips on programming style, efficiency, etc; Language and interface enhancements in APL; working products; How J/APL is useful; Making APL/J useful to end users (as opposed to developers); Success stories using APL/J as well as unsuccess stories; Vendors cooperation in research for improving APL." Q22: Other comments or Suggestions? In the respondents' own words: "Keep up the good work; Thank you Committee! Ryerson's facilities, especially classroom were much superior to what was available in past at U.T.; It would be great to have a participants list in the registration kit at the beginning conference. Always is it good service (many other Conferences use it and I enjoy it); Encourage presenters to have a script beforehand. Some of the projections via overhead was unreadable. Need to schedule beforehand. Great conference. Thank you; Allow a 1.25 - 1.5 hours for lunch. Lunch is too short (several comments); Ran out of coffee/refreshments during break; Great Conference, all the organizers deserve a lot of credit; More interesting content than any recent conf; Thank you for a great conference; Thanks for your effort in sponsoring APL97. It was certainly one of the best ever; Overhead projections not readable in 90% of the workshops, Sit at the back of the room to test for yourself; I compliment Toronto SIG on a very well done conference, including location and accommodations, and program pamphlet; Advanced applications are nice, but for the average programmer, the problem is: 1) ease of reading code, 2) debugging it, 3) with multiple programmers, how to keep their work organized; Some workshops were less useful as all we did was typing ... creating images in J which left us some assignment. We were too busy typing to follow the message/contents of the workshop. Suggestion: reduce the number of terminals and operators; Good Conference; An excellent conference. The program committee should be commended for a job well done." File APL97SRV.ZIP containing all analysis fns and APL97 survey data, is downloadable free from the BBS\APL at 703-528-7617, in File Area FREE. APL97SRV.ZIP has also been sent to Toronto SIGAPL and Waterloo for electronic distribution. References: 1) Dick Holt, "APL96 Conference Survey Results", APL Quote Quad V27, No. 1, September 1996, p 36-38 2) Dick Holt, "1994 SIGAPL Survey Results", APL Quote Quad V25, No. 3, March 1995, p 9-15 Dick Holt can be reached at dholt@CapAccess.Org, fax 703-528-7617, or mail at 3802 N. Richmond St. Arlington VA 22207 USA