START's administration console is the "central command center" for your conference. Its clean, single-panel interface provides all the controls you'll need to manage your entire submission and review process. The administration console includes the following features:

Conference Setup:
This is where you enter basic conference information, which customizes the START interface for your event. You can enter the name of the conference, its acronym, meeting dates, location, the conference homepage URL, the name of the PC chair (used to auto-sign email sent by START), and the main contact email address for the event (which will appear on the "from" line in email sent by the PC chair). There is also an option to enter a submission deadline - after which newly arriving papers will no longer be accepted by START.

Submission Page Setup:
This tool automatically builds your submission page, and initializes the server to handle on-line submissions. The first step is to enter the document formats which you plan to allow (PDF, Postscript, MS Word, RTF, Powerpoint Presentation, and/or Plain Text, etc.). You can also determine whether START should do document conversion for you (among the formats supported). You can also determine whether or not you plan to run a blind review process. If so, the submission form will be generated to include special instructions for authors (e.g., not to include their names on their submitted manuscripts, etc.). The "blind review" selection will propagate throughout the review process; in other words, reviewers will not be able to see the author names, until you decide to expose them. This tool also allows you to add submission categories and keywords to your on-line submission form - which will then be selectable by the authors. If you have multiple submission types - e.g., full papers, short papers, demos, panel proposals, etc. - you can create a category variable to reflect these different formats. You can also create a category variable to characterize the manuscript's subject area; this feature is particularly useful if submissions from different technical areas are reviewed by semi-autonomous sub-committees. The category information can be supplemented by keywords, which can be checked by submitting authors. These keywords are often used by committee members to "bid" on the papers they wish to read.

Monitor Incoming Submissions:
Using this tool, you can check on the progress of your submission pool, as papers get sent it. Two views are presented - a brief overview list of submissions, and a more comprehensive report (which includes summary, category and keyword information, as well as links to download the actual manuscripts).

Delete/Merge/Renumber Submissions:
START provides a password-protected login entry for authors who wish to revise their submissions. Occasionally an author bypasses this option, and resubmits a revision as if it were a new paper. You can use this tool to merge the information from duplicate submissions, and to delete the redundant versions. Paper deletion is also necessary when an author decides to withdraw his/her submission from consideration.

Edit Submission Information:
Occasionally, editing the submission information is necessary. For example, authors occasionally mistype their own email addresses on the submission form. This tool allows you to change any of the information in a recorded submission. You can even upload a revised manuscript, on behalf of the author.

Get Author Information:
This tool extracts submission information from the START database, and presents it in a plain-text form. Check-off boxes are presented to select the type of information desired. For example, if you would like to generate a mass-mail list of submitters, you can request "author emails" - and the result will be a textual list of email addresses, ready to plug into any email client program. Any of the submission information can be selected, and retrieved in a textual format.

Setup Reviewer Policies:
This tool sets up the structure for your evaluation and review process. The options presented control the degree to which you share information with your committee, and when you decide to share it. The selections can be altered at any time in your review process. You can return to this page at key points, and update the policies to reflect the current stage of your editorial workflow. Some of the options available are the following:

Access to Submission Information: This policy controls the degree to which the committee can view information on the submission pool. Available options are:

  • Committee members can view the submission list, as well as all of the papers.
  • Committee members can view the submission list (including the short summaries), but papers are restricted to assigned reviewers.
  • Committee members cannot view the complete submission list. Members will only see information about the papers they are assigned to review.


Bidding for Papers: START can be of great help in assigning reviewing duties, if PC members "bid" for papers to review. In the bidding phase, reviewers inspect the summaries/papers which were submitted. Then they send in their preferences, selecting one of the following "bids" for each submission:

  • I want to review this paper.
  • I am qualified to review this paper.
  • I don't know the area that well.
  • I have a conflict of interest on this paper.


Using these "bids", START will help you automatically generate a first-draft review assignment (which you can adjust manually). Hence the available policy selections are:

  • Enable Bidding.
  • Disable Bidding.

The bidding policy can be turned on at the beginning of the "preference-selection phase", and turned off when you make your review assignments.

Unsolicited Reviews: Obviously a paper's assigned reviewers are expected to complete their reviews. However, you can optionally allow committee members to voluntarily submit reviews for any paper - except those who are identified as having conflicts of interest. It is often helpful to have more reviews (rather than fewer) - and in any case, you can always delete reviews which fail to contribute additional useful information. The options you have for this policy are:

  • Any member can review any paper, excluding conflicts.
  • A submission can only be reviewed by an assigned reviewer.

Sharing Reviews: This option allows a reviewer of a paper to see all other reviews of the paper - after he/she has submitted a review. (Otherwise, there may be a temptation to just copy someone else's review, and not read the paper.) You can do this at any time in your review process. It is particularly helpful for committees which do not have formal meetings, since it helps achieve consensus beforehand. The options available are:

  • After a member completes a review for a paper, he/she can see the other reviews for the paper.
  • Committee members cannot see the reviews written by other members.

Report Access: After the reviews are completed, you may wish to give your committee access to the review information, in one form or another. If your conference committee is not meeting (physically) to determine the program, the report facility will be invaluable. Even if you are hosting a real meeting, the reports can still prove useful, by letting members achieve some consensus opinions beforehand. You can control the degree of information you wish to share with your committee, and when you wish to do this. There are two types of reports you can show your committee, the Summary Report and the Comprehensive Report. The Summary Report includes paper titles, and some statistics based on the numerical scores awarded by the reviewers. The summary report does not include reviewer names, nor does it include the detailed comments in the reviews. The available options for setting access to the Summary Reports are:

  • Allow everyone to see summary information for all papers.
  • When a reviewer requests the summary report, exclude papers where there is a conflict of interest.
  • When a reviewer requests the summary report, only include summaries for papers which he/she reviewed.
  • Publish no summaries at all.

The Comprehensive Report presents all information in the reviews, including the reviewer names, the numerical scores, the detailed comments, and "confidential comments" (for the use of the Program Committee, but not sent to the authors). The policy settings for accessing Comprehensive Reports are:

  • Allow everyone to see reports for all papers, excluding conflicts of interest.
  • Only allow reviewers to see reports for papers they reviewed.
  • Publish no reports at all.

Message Boards: You have the option to enable message boards for each paper. This selection creates "webmail" threads for each submission, in which reviewers can discuss the merits/liabilities of including the paper in your program. Access to a submission's message board is limited to reviewers who are able to see the comprehensive report for the paper (this includes the program chairs). There is also a "main message board", in which all PC members can participate. The messaging facility can be of great help in running an on-line PC meeting - and it can also help build consensus among the committee before your actual "submission vetting process." However, whether or not you enable this feature is voluntary - and it can be enabled at any time during your review cycle.

There is also an option to forward all message-board postings as regular email. The email will be sent to all the people who have access to the message board used - and it will include a back-link to the message board's URI. The email's "From" address will be that of the person who posted the message - and all recipients are included on the "To" address line (so follow-ups can automatically be sent to the same recipient group). In this way, discussions about papers can be initiated within START; however, they need not remain there. With email forwarding, a discussion thread can be initiated within START, and then continued via regular email. Again, this feature can be optionally enabled or disabled at any time.

Multi-Track Conference Management:
START V2 includes comprehensive support for multiple tracks and hierarchical reviewing, so that sub-editors can assign their own sub-committees, and control their processes via a mini version of the START manager console.

There are many ways to use tracks. Some conferences create tracks for various submission categories. These can be defined by technical areas, or submission type (short papers, papers, panel proposals, etc.) - or, tracks can be defined by both subject area and submission type. Some conferences create a track for each committee member. Then each member can recruit his/her reviewers autonomously, to review the submissions assigned to the member. Some conferences even create a separate track for every paper.

Other reasons to use tracks are not so obvious. For example, you may want to recruit external reviewers. These are not members of your committee; rather they could be other specialists, whom you call on to review a few papers. In such a case, you could set up a track called "Reviewers". Then all papers requiring external review could be inserted in this track. The track policies can be set at the most restrictive level (in which reviewers are only aware of their own assignments, and their own reviews). Then you could create "Reviewer Track" accounts for the external reviewers, with their assignments made using the "track manager" console. The reviews will still "flow up" to the main level.

Tracks can also be used solely for organizational reasons. For example, it may be useful to create reports falling into various "decision categories", e.g. ("destined to be accepted", "destined to be rejected", and "borderline"). These tracks could even be dynamic - i.e., you can move the papers from one category to another, as your "vetting process" proceeds. You could, if desired, make your entire committee members of these tracks - in which case they could participate in the progress of the vetting process. There are many tools at the "top level" to manage this sort of activity; however, you may (or may not) also want to use the partitioning mechanisms provided by the track facility. And so on. The main points are (1) tracks can be used solely for organizing your submission pool into various groups; and (2) the paper-to-track assignment is not static; it can be changed at any time, without disturbing other information in the database. It is also nondestructive - assigning a paper to track does not remove it from the "main" submission pool.

Tracks are established using the following four steps (the tool screens are selected from the manager console).

Create Tracks: First define the track names, e.g., "Posters", "Invited Talks", "Regular Papers", "Short Papers", "Student Presentations". After pushing an ENTER button, the track sub-domains will be activated - each with its own web space for managing information within the track. At any time in your editorial process, old tracks can be deleted, and new tracks can be added (without affecting the information in the other tracks).

Appoint Track Chairs: You can appoint a "track coordinator" (and up to five assistants) to manage the workflow for a track. A track chair can also be one of the PC chairs; in fact, the PC chairs are automatically allowed to login as track chairs. The track chairs get access to mini-versions of the START manager console, for the express purpose of handling their tracks.

Map Submission Categories/Keywords to Tracks: You can set up a "profile" for each track - in which the author-inserted categories and keywords will help "guide" a submission into the track associated with those categories and keywords. Then, the track assignments can be tuned with the Paper Partitioner.

If this tool, one first decides whether to use categories or keywords to build the track profile. In the case of submission categories, one can determine which values of the category variables should be associated with each track. Then, in the preliminary submission-to-track assignment, the appropriate track for each submission is determined, and the submission is copied into that track. (Then this assignment can be adjusted, if desired.)

Keyword-defined tracks are established as follows: For each keyword, you are requested to consider a hypothetical submission, written on a subject characterized by the keyword. Then you can determine the track in which the hypothetical submission should be placed. If a submission contains multiple keywords, associated with different tracks, START makes an "educated guess" in deciding where it should be placed. The track assignment can then be adjusted manually, as with category-defined tracks.

Note that creating "track profiles" (based on categories or keywords) is not mandatory. You can create tracks using your own "conceptual track profile" - based on any criteria you have in mind. Then you can manually insert submissions into the tracks, to suit your own purposes. A submission can be included in multiple tracks at the same time, and it can be moved around between the existing tracks.

Partition the Papers: This tool allows you to edit the placement of submissions in tracks. The main window shows the current allocation of papers to tracks. Editing can be done via sub-windows, presenting a "track view" and "submission view", respectively. The "track view" tool lets you add/delete submissions from a particular track. The "paper view" tool lets you concentrate on any particular submission - in which you can change its track associations, as necessary. A paper can be included in any number of tracks - or rather, it need not be included in any of the tracks.

Manage Administrator Accounts
This tool lets you add other administrators for your conference, such as co-PC chairs, webmasters, etc. These people will have the same access as the main manager - except that they cannot add or delete other administrators' accounts. That is the sole prerogative of the main START manager.

PC Scratchpad:
This tool helps you form your committee, if it is not already established. The PC Scratchpad is a long spreadsheet, in which you can enter the names of colleagues who are qualified to serve on your committee. Then, from these names, you can incrementally invite people to join, until you realize your ideal program committee. For each "candidate", you enter the person's name, last name, greeting name (i.e., how you wish to address the person in email), and his/her email address. If you decide to invite a person to serve on the committee, you can check a box (denoted "Invite"), which puts the person on an invitation list. Then the invitations are sent out using START's Invitation Tool. After you have invited someone, two new check marks appear next to the person's name - "Accepted" and "Declined". For the people who accept your invitation, their information is copied into the PC Account Creation tool - so that you can easily create new START accounts for them.

Invitation Tool:
The invitation tool contains a draft letter to invite PC members. It can be edited as you wish (and the contents may be saved). Under the letter is a list of "candidate PC members" who have yet to be invited. Check-off boxes are provided to denote the people you wish to invite in one mailing. In the actual letter sent to a PC member, his/her greeting name is entered automatically (as it was entered on your PC Scratchpad tool). All mail will be sent from you (the PC Chair), with your email address on the return path. Each potential PC member will get an individual copy, addressed to him or her. All you need to do is to press one SEND button.

Committee Account Manager:
This tool presents a split-screen interface. On the top, there are fill-in slots to create new Program Committee Accounts. Fields are provided to enter a member's username, password, email address, first name, last name, and the "greeting name" to be used in auto-generated correspondence. If you use the Scratchpad Tool to handle your invitation process, the people who accepted your invitation (but do not yet have accounts) will appear at the top of the list (with pre-formatted fields for their contact information, taken from the Scratchpad). If you choose not to use the Scratchpad, then you will have to enter the contact information in the empty slots provided for new accounts. (In such a case, visiting this tool would be the first step in creating PC accounts.) The bottom-panel of the tool presents a list of existing accounts for the Program Committee. The fields can be edited (to change any of the information for a member); also, there are check-boxes to delete an account altogether.

Send PC Members Account Information:
This tool presents another split-screen view: on the top is a draft letter to send to your program committee members, informing them of their START accounts for your conference. On the bottom is a check-off list of people to send this information; a member whose name is checked will receive a personalized copy of the letter. The letter can be edited to suit your needs. The draft (which initially appears in the window) is a reasonable starting point. It contains a standard greeting; it informs members about their accounts; it requests them to make a "first contact" with START, to update their profiles; and it contains some standard instructions about reviewing protocols, etc. The draft letter is meant to serve as an example, which can be edited to suit the conference's needs. The check-list on the bottom allows you to select the people who will receive this mail - which will be personalized for each recipient.

Correspond with Program Committee:
This tool lets you send ANY message to members of your committee. Again, it presents a split-screen view - a prototype letter on top, and a check-off list of recipients on the bottom. The letter can be used for any communication with your PC, for example, to notify them about access to review reports, to send information about the program committee meeting, or whatever. The letter can be edited however you wish (in a text box); also, you can optionally use a text letter from a file which you drafted using another tool. Then it gets uploaded to START, and sent to all of the selected PC members (with the information personalized for each).

Initiate Bid Process:
This is another mail tool, to inform your committee about the "bid process" - in which they can send in their reviewing preferences. (This is an optional step, which is not necessary if you plan on selecting all review assignments for your committee, regardless of preference.) The page is structured like the other mail tools: there is a split-screen view, with a draft letter on top, and a check-off list of recipients on the bottom. The draft letter contains some general guidelines for bidding on papers; however the actual text can be edited to suit your needs. This letter is sent after you "enable bidding" (see the section on policies above).

Monitor Bid Responses:
Here you can monitor the status of responses to bids. Again, if you want to use the START tool to help assign papers, it is important that the PC members send in their bids. The page contains an alphabetized list of a list of committee members (i.e., people with committee accounts), along with information on when (or if) their bids were received. If you also have some " quasi-official " members - whose accounts were established to help you with the process - you obviously need not worry about their bids. On the other hand, for those who review papers, bids are an important first step. In some situations, you may have to remind these people personally, by email or via some other means.

Make the Review Form:
START allows you to configure your review form to suit your conference's needs. Unlike other review tools, START will let you enter any review criteria you wish - and numerical scores can range between any minimum/maximum points. Based on the form fields you enter, the actual review form will be automatically customized for your event.

Almost all review forms possess two parts. One part is structured like a "multiple choice" exam. In this part, the reviewer quantitatively ranks the submission, along the guidelines which you set up. The other part of the review consists of written comments, which are meant to justify the "multiple choice" scores. All START review forms contain two boxes for written comments. One part is sent to the authors after the decisions are made; hence, this part should contain the bulk of the commentary regarding the submission. The other box is for comments to be held "in confidence", for the private use of the committee.

Regarding the "multiple choice" portion of the review form, START allows you to introduce any "variables" you wish. For each item, you can decide whether the scores should be quantitative or qualitative. For a quantitative item, you can fill in a minimum and maximum, denoting the scoring range for that item. For a qualitative rank, you can enter ascii text in the box associated with the item. Each line in the box will turn into "selection" with a scrolling box, and will appear on the review reports accordingly. Common examples of "quantitative variables" are:

  • Originality of the Work
  • Technical Correctness
  • Overall Quality

and the like. It is essential that at least one category like this is scored numerically, enabling some statistical analysis to be made over the range of papers, and to sort them according to some key index (such as Overall Quality).

The "multiple choice" portion of the review may also contain qualitative (non-numerical) rankings, e.g.,

  • Suggested Presentation Type (paper, poster, abstract)
  • Nominate for Best Paper Award (no, yes)

These are just examples. In reality, the items you choose to enter are totally up to you. There's no limit to the number of items you wish to include on your review form (however, it is suggested to make it no longer than necessary). Likewise, there is no limit to the scoring range. For quantitative areas, you can choose any numerical range you desire. For qualitative variables, you can insert as many choices as you want.

After you press the ENTER key, your review form will be created. You can proofread it, make changes, etc. - however, all editing of the review form should be done before the actual review process starts.

Check Review Form:
This tool selects a random submission, and shows you the review form for that submission. The review form pre-formats the submission's information (e.g., its title, paper ID, etc.), and it also auto-formats the reviewer information. When you select this test review form, your name will appear as the reviewer. Using this draft form, you can preview the actual form (as it will appear to your reviewers). If you wish to make changes, you can return to the review form editor, and re-edit the form there.

Assign Reviewers:
This page contains a set of tools which help you assign submissions to reviewers on your committee. If you plan on making all the assignments manually (without taking into account the reviewer preferences), then you can proceed directly to the assignment phase. However, if you requested your PC members to send in bids, START's automatic assignment algorithm can help you greatly - at least in making a first-draft assignment. Then you can evaluate (and manually tune) this allocation. Before running the auto-assigner, there are several options which need to be reviewed. These will have a large effect on the results of the assignment.

Number of reviews for each submission. You are requested to enter the number of "desired reviews per paper." This number is used by START to "seed" the assignment algorithm, which will attempt to satisfy this quota based on the reviewer preference forms. Aside from its use in the assignment algorithm, this number is not used anywhere else in START. It does not limit the number of reviews you can receive for a paper; likewise, it prevent you from manually adding additional reviewers. Also, this number can be changed at any time. For example, you may wish to make your assignments incrementally. In a "first round", you could get two reviewers per paper - and then, if you are satisfied with the results, you can add more reviewers in a "second round", etc. In this case, you would increase the "number of reviews per paper", and then re-run the assignment algorithm. The old assignments will remain the same (unless you decide to change them). The higher "target number" will just add to the existing assignments.

Bid Adjustment. The assignment tool also displays an alphabetized list of your reviewers, with each name linked to the reviewer's bid sheet (or preferences). You can view (and edit) a reviewer's assignment preferences by clicking on his/her name, and then re-submitting the person's preference form. You may have your own opinions regarding the reviewer's strengths/weaknesses. These opinions may differ from the reviewer's self-evaluation. Also, if a reviewer failed to send in his/her bid form, this is the place where you can enter the information yourself, based on your knowledge of the person's expertise. The automatic assignment algorithm is very sensitive to the reviewer preference information (it chooses reviewing assignments based on these bids). Hence, if the auto-assigner is to be used, it is important to ensure that bids are entered for all reviewers, and that the reviewers' self-assessments correspond to your assessment of their abilities. If someone failed to enter a bid, his/her name will be highlighted in RED, for easy identification.

Quota Adjustment. The auto-assignment tool processes reviewers and papers, optimizing the reviewer preferences with your target number of reviews. Hence, as a default, your reviewers will end up with more or less the same number of submissions to read. However, you may have some reviewers who requested a "light load", i.e., they requested fewer paper assignments than other reviewers. If you do not wish to assign the same number of reviews to everyone, you can hand-adjust any reviewer's "quota". The quotas can range from 0 (no assignments) to an arbitrarily large number. (This means that the reviewer will continue to receive assignments, until the "Reviews per Paper" is exhausted for all papers. This is the default, if no quota is given to a reviewer.) If a reviewer did not send in a bid form (or you did not enter the information), his/her review quota will automatically be set to 0. The assignment algorithm will not allocate papers to reviewers whose preferences are unknown. Quotas are also useful for Program Chairs (and webmasters). Such people may be overseeing the process - but may not want to review any papers (or in the case of webmasters, they may not be qualified to review any). In such a case, these people can have their quotas set to zero, ensuring no auto-generated assignments. Again, assignments can be manually made despite the quota information. This is a general policy in START: no hand-entered decision will ever get overridden by the system.

Make the Assignments. Assignments can be auto-generated (and manually entered) via a choice of three tools. All three have the same functionality; however, they present different views of the data. You can choose the method which is most suitable for you. Also, you can use these tools interchangeably - making some assignments with one tool, and other assignments with another tool. The information and functionality will be the same - however, the presentation and entry modalities are different. The three "views" into review assignment-making are the following:

  • Scrollbar Spreadsheet: The information is in a tabular format, in which each cell contains a paper title, its authors, and scrolling lists to select the reviewers. Each scrolling list is pre-formatted, and contains reviewer names. A "currently assigned" reviewer will always be listed as a pre-selected option in one of the boxes, corresponding to a submission. This tool is very convenient; however, it tends to get a bit unwieldy when you have a lot of papers, and/or a relatively large committee. In such a case, one of the other methods should be used for assigning reviewers.
  • Checkbox Spreadsheet: This spreadsheet presents the same information as the scrolling-list table - however, it is represented differently. Each paper title is listed in a table, along with check-boxes for every member of your committee. If a member has been assigned to the paper, his/her box will be checked (and otherwise, it will not be checked). For a conference with many papers - where the committee is relatively small - this may be the easiest way to view/enter the information.
  • Column-Separated Textual Spreadsheet: This is by far the most compact way to enter reviewers. This tool presents a purely textual view of the assignment duties, where the reviewers are designated by their userids in START. It is a basic column-separated spreadsheet, in which each paper occupies one line, and the reviewer usernames are delimited by ascii separators. If you want to analyze your assignments using a program like Excel, you should use this tool to get a "dump" of the assignment information - and also, to upload a new assignment (if desired).

 

Regarding functionality, all three assignment tools work the same way. The following options are offered on each tool:

 

  • GENERATE ASSIGNMENTS: Run the auto-assigner, using the bid and quota information entered for your reviewers. Previously saved assignments (which may have been entered manually) are always conserved and merged into the new assignment.
  • SAVE THESE ASSIGNMENTS: Save the reviewer assignments which appear on the page, either after running the auto-allocator, or after manually entering some assignments, or both. Whatever appears on the spreadsheet will become the new reviewer assignment.
  • RESET TO SAVED ASSIGNMENTS: If you ran the auto-allocator after making some manual assignments - and you are not happy with the result - then you can use this button to "rollback" to your previous allocation.
  • DELETE ALL ASSIGNMENTS: Start the assignment process all over again. All current allocations will be deleted.

As for the assignment algorithm, it always attempts to make a fair distribution, by taking into account everyone's preferences, excluding conflicts, and assigning papers on a round-robin basis. The round-robin scheme is ALWAYS over-ridden by the current set of fixed assignments, made by you.

Even before running the auto-assigner, you may wish to manually assign some papers. This is particularly useful when some submissions are on arcane subjects, and only a few of your members are specialists in those areas. In such cases, it's a good idea to manually make these assignments before running the algorithm. Then, the auto-assigner will assume that these assignments are fixed - and at the same time, it will try to give all reviewers (who don't have quotas) the same number of reviews. In most cases, reviewers are satisfied with their assignments - since, in the selection process, the papers are given out in the order of their bids. Moreover, a reviewer who has a conflict-of-interest with a paper is never assigned that paper to review.

START the Review Process:
This is another mail tool, used to inform your reviewers about the review procedures. It is structured like the other mail tools: there is a split-screen view, with a draft letter on top, and a check-off list of recipients on the bottom. The draft letter contains information about where the reviewers can find their assignments (they are on a "To Do" list, activated when a reviewer logs on). The letter also contains some fairly generic guidelines for reviewers to follow; however the actual text should be edited to target the needs of your conference. This letter should be sent after you make your review assignments. Personalized copies will be sent to each reviewer (who is selected to receive this information.) Typically, the reviews start arriving shortly after this letter is sent out.

Monitor Review Progress:
This tool allows you to monitor the progress of your reviewers. The screen presents two tables. The table on top lists all of the submissions which are subject to review; each submission lists the names of the reviewers assigned to review it (as well as the names of "unsolicited reviewers", if any exist). There are three options associated with each reviewer's name (for a submission):

  • View Review: If the review is completed, the reviewer's name actually be a hyperlink, which will let you access the review. (If the review is not done, no hyperlink will be presented - since there's nothing to see.) By clicking on the reviewer's name, you can see his/her review for the submission.
  • Edit Review: You may also edit a particular review. To do so, you need only click on an EDIT button next to a reviewer's name (and under the submission's title). If no review was completed by the assigned reviewer - and if you wish to enter one on his/her behalf - you may do so with this tool.
  • Delete Review: In the case where you wish to delete a review (for a particular reviewer and a submission), you can click on a DELETE button next to the reviewer's name - and the review will be deleted from START's database.

The table on the bottom of this page organizes the same information in a different way. The table presents an alphabetized list of review names, each of which is linked to the reviewer's personal "To Do" list. When you click on a reviewer's name, you are shown a list of the submissions he/she has already reviewed, and those which have not yet been reviewed. Under each title on the "already reviewed" list, you are given the same options as above: you can view, edit or delete the review in question.

Report Generator:
START provides three ways to summarize the review information, to help make decisions on paper submissions. This toolscreen gives a description of each type of report, along with buttons to retrieve each. The three types of reports are:

 

  • Unsorted Summary Report: This is a simple report, summarizing the "multiple choice" sections of the reviews for all papers. For each paper, the report presents some statistics over the numerical scores awarded by reviewers of the paper, e.g., minimum, maximum, average, and variance (with respect to the average). This report is ordered by submission number.

 

  • Sorted Summary Report: This report presents the same information as the Unsorted Summary; however, it is ordered by "relative quality" (as indicated by the numerical scores). This report will show you - at a glance - which papers are "destined to be accepted on your program", and which are "destined to be rejected". You can then concentrate most of your energy on the submissions which fall into the "maybe" category - i.e., those over which there will be some debate among your co-Chairs or committee members.
  • Comprehensive Report: The comprehensive report presents all the review information for all the papers - including the detailed comments and the reviewer names.

The submission listings in these reports are all linked to additional information about the papers - for example, the abstract, keywords, categories, authors, and the manuscripts themselves.

Print Reports for Meeting:
START lets you generate an individualized report set for each member of your program committee. The "report folio" will contain all three types of reports - and the contents will adhere to the policy guidelines you stipulated (in the policy tool) for reviewer access to information. Each "combined report" is in PDF-format, with indexed hyperlinks to access each sub-report, and to jump to the information for each paper within the comprehensive report. The main benefit of using the START-generated reports is obvious: no conflicts of interest will be compromised. That is, if a reviewer has a conflict-of-interest with a submission, then his/her comprehensive report will not contain the reviews for that submission. You can then bring these PC-reports to a program committee meeting, and give each member a personalized report-set.

There is obviously another way to do this, which could prove to be easier (depending on how you run your meeting). Shortly before your meeting, you can revisit the Policy Tool, to set the report access policies appropriately (as you want them for the meeting). Then you can ask your PC members to download and print out their own reports - and then to bring them to the meeting. This solution would amortize the printing costs among your committee. It also can lighten your (physical) burden, since you wouldn't have to carry stacks of reports for all members. Also, people with laptops won't need any hard-copy reports at all: they can save the reports on their laptops, and use these versions for perusal during the meeting.

However, if you want to ensure that everyone has a hard-copy report set, the START report generator will make your meeting preparation a lot easier.

Make Decisions:
This tool presents a list of all submissions, with check-boxes and "status fields" to enter your final decisions regarding their inclusion in the program. The two main choices for each paper are the most important - accept or reject. The default for all submissions is "Reject" - which means you only have to alter the status of submissions that you decide to accept.

Often a conference accepts submissions in a variety of submission categories, e.g., Poster, Short Paper, etc. Sometimes, the manuscripts are submitted in these categories (via selections on the submission form). Other times, submissions are generic - and it is up to the committee to decide on the type of "acceptance" it wishes to grant. START allows you to enter any conditions you want for the accepted papers. Next to each submission entry, there's a blank field for "conditions" regarding the acceptance. You can fill in any information you want in this field. It will be stored, and will appear in the mailtool for author notification. Then, you can selectively send mail to authors of these papers, based on the conditions you note here. The job of writing the actual letters - explaining the conditional acceptances - is yours. Also, if you accept various categories of submissions (e.g., posters, short papers, long papers, student presentations, demonstrations), then you can draft one letter for each type of acceptance - and then send these letters in batches, corresponding to the types of acceptances.

Author Notification - Rejected Papers:
This mail tool is similar to the others described above. A default letter appears in a box, which you can edit and save. The list of rejected papers appears at the bottom. You can selectively (or simultaneously) send these letters, and the relevant data will get plugged in. In all the author notification mailtools, the paper's review information (including the non-confidential comments) is automatically appended to your note. 

Author Notification - Accepted Papers:
This screen is formatted just like the other mailtools, with a draft letter on top, and a list of (accepted) submissions on a "to send" list. Next to each submission entry are the "conditions" (if any) which were associated with the acceptance. This notation can help you draft your letters. The acceptances can be sent out in groups - one group for each "acceptance condition", e.g., for posters, short papers, full papers, invited talks, etc. Then, each letter will be personalized for the authors.

Correspond with Authors:
This tool lets you send any message to the authors of accepted submissions. You can use this mail tool to inform them of deadlines for final submissions, formatting guidelines, or for any purpose you want. Again, the tool presents a split-screen view - a prototype letter on top, and a check-off list of recipients on the bottom. The letter can be edited however you wish (in a text box); also, you can optionally use a text letter from a file that you drafted using another tool. Then it gets uploaded to START, and sent to all of the selected authors (with the information personalized for each).

Advance Program Maker:
This tool creates several views of the accepted submissions, which help facilitate the following (final) tasks in your review process: creating an advance program (for pre-conference publicity); generating a textual list of accepted submissions, to paste into a mass-mailed "call for participation"; and making a list of "author contacts", typically used by third-party publishing houses (or an internal publishing chair), to finalize arrangements for the proceedings. When you turn to this tool, the following links will be presented:

List of Accepted Papers: An HTML list of accepted submissions is generated - with each title linking to the abstract for its submission.

Archive of Abstracts: START packages the "accepted paper list" (and all the abstracts) in a tar archive. The directory included in the archive can then be relocated to your main conference website, to help publicize your program. Within the summary file, the abstracts are unanchored - meaning that this directory can be moved anywhere. As long as the summary file is in the same directory as the abstracts, people will be able to view them - from wherever you decide to put them.

Textual List: For mailing out conference information, we also provide the accepted paper list in text form. You can just copy this textual list, and paste it into your mass-mailed Call for Participation.

Contact Information: Conferences often use third-party publishing houses to generate the proceedings. (If your conference is sponsored by a major professional society, their publishing house probably takes care of this job.) In such a case, the publishing house usually contacts the authors about manuscript preparation, formatting, deadlines, copyright waiver, etc. In such cases, the publisher needs a list of papers in advance, with author contact information. START generates this list for you, in a style accepted by most sponsor organizations. You just have to forward this list to the editor in charge of your proceedings.

Spreadsheet Maker:
This is a relatively new addition to START V2, which clients have found to be very useful. The Spreadsheet Maker extracts collections of records from the START database and downloads them in the form of Excel spreadsheets. The downloaded file can then be imported into Microsoft Excel, Access, and many other client spreadsheet/database programs. Note: This information is already accessible in START, via the formatted reports which START provides. However, you may wish to reorganize the data with a different view, or import it into your favorite spreadsheet/database program. If so, this is the tool to use. Using the worksheets produced by the Spreadsheet Maker, you can process the data however you want, and generate any type of report imaginable.

Each spreadsheet has the following form: The first row contains the field headings for the extracted records. Each subsequent row corresponds to a data record, with the cells corresponding to the fields within that record. The following views can be exported as Excel spreadsheets (i.e., as files with type xls).

Submission Information: This tool lets you create a spreadsheet containing the information for all the submissions. You can choose the "view" you wish to see, via scrolling menus (each of which contains all the submission form fields). Using these menus, you can decide which items to include in the generated spreadsheet, and the order in which the fields will appear in each row. Any data can be requested, except the manuscripts themselves (which are too big to fit in a spreadsheet).

Bid Information: This spreadsheet generates all the bid information which was collected for your conference. The rows correspond to papers (referenced by confirmation number). The columns correspond to your reviewers (their names are listed in the column headings). Each cell contains the reviewer's bid for a particular paper.

Assignments: You can also make spreadsheets that contain your review assignments. You can select from several options to determine the type of information you wish to see, and how you want to see it. The options are as follows:

  • Index Method [By Submission/By Reviewer]: If you select "Index by Submission", the first item in a row will be a Paper Number, followed by a list of the reviewers assigned to that paper. If you select Index by Reviewer, the first item in a row will be a reviewer's name - followed by a list of papers assigned to the reviewer.

 

  • Include Track Assignments [Yes/No]: If your conference is segmented into tracks, you can also include the assignments made at the track-level (if there are any). If your conference does not have tracks, this selection has no effect.

 

  • Include Bids [Yes/No]: You can optionally include the bid information (if you used bids). This allows you to compare the result of your assignment to the bids made by the reviewers. If you index by submission, the reviewer's bid for a submission appears next to his name, in parenthesis. If you index by reviewers, each assigned paper number is followed by the reviewer's bid for that submission, in parenthesis.


Reviews: You can also generate a report containing all the reviews, for every paper. Each record represents one review for a paper, and rows are ordered by Paper Number. You can choose the view you want to appear in your spreadsheet, via scrolling menus (each of which contains all the review form fields).