AO3, GDPR and You

http://archiveofourown.org/admin_posts/10518

Outlines of a man and woman speaking with word bubbles, one of which has the OTW logo and the other which says 'OTW Announcement

You may have heard that the European Union's new regulation on data protection, the General Data Protection Regulation (GDPR), is coming into effect on May 25. It has effects on sites and organizations all around the world, including Archive of Our Own.

The GDPR's primary focus is on data privacy and transparency, and on giving users more control over what their data is used for. The AO3 team has been working hard to comply with all GDPR requirements, and to make more information available on how we use your data across the site. Here's what you need to know about upcoming changes.

Terms of Service

We've recently updated our Terms of Service. In this update, among other changes:

  • We added more explicit information about how we use the data you provide to offer certain features—such as keeping track of where you've already left kudos—and to display the content you provide in your works, comments, and profile pages.
  • We added a new Age Policy (more on this below).

A summary of all changes and a link to the full document can be found in the Terms of Service update announcement. Everyone who uses AO3 will be required to agree with the new Terms.

Age Policy

We have created a new Age Policy, which is described in our updated Terms of Service. This restricts how old you must be in order to create or own an AO3 account.

While the general minimum age to create an AO3 account remains at 13 years old, there are specific requirements for those who reside in the EU. If you are under 16 years old and live in the European Union, please check your country's current age of consent for data processing.

You must be of age to give us consent to use your data in order to use the Archive, or else we can't legally, for example, store your kudos or comments (even as a guest). Our Policy & Abuse team will contact users if there are reasons to believe that they are under their country's age of consent for data processing, and may delete their accounts as needed. Now is a good time to review your profile and works for any outdated information about your age!

We're sorry for any inconvenience this causes any European users who will now be under the age of consent in their country. We'll be glad to welcome you back when you're of age to use AO3 once again!

Consent

Starting soon, a banner will pop up requesting that you accept our new Terms of Service. It will highlight some important aspects of the data processing we do here on AO3 regarding your works, comments and so on. That way, everyone can provide informed consent when they accept the new Terms.

An important note about this: According to the GDPR, information about personal relationships, religious beliefs, political affiliations, and sexual orientation is defined as "Special Categories of Personal Data". We know our users can sometimes mention information related to this in their profiles, works, notes, tags, comments, bookmarks, and so on. Please be mindful of your privacy when entering this information in publicly-accessible pages. Any information you publish in openly accessible spaces will be publicly available to everyone.

Above all, AO3 remains dedicated to being a place to preserve and share fanworks. We're proud to offer this service by fans and for fans, in a format that is free and lets you retain full control of your works. We don't display ads or monetize user data in any shape or form and don't ever intend to.

Learn More

For more information on the GDPR, please check out The European Commission's Rights for Citizens or Wikipedia's information on the GDPR.

If you'd like to know more about how AO3 and the nonprofit behind it, the Organization for Transformative Works, have prepared for GDPR, please contact the OTW Legal team. If you have any specific questions on how these changes affect you or the AO3, or if your native language isn't English and you need some assistance to understand the changes we've made, please feel free to get in touch with our Policy & Abuse committee.

If you have any questions about these upcoming changes, please feel free to leave them in a comment below.

Downtime Report: May 10-14

http://archiveofourown.org/admin_posts/10515

Following our May 10 deploy, the Archive experienced a number of issues stemming primarily from increased load during the Elasticsearch upgrade process.

As we noted in our March downtime post, the Archive hasn't been running at full strength due to this upgrade. Compounding the issue, it has taken significantly longer than planned to get the new code deployed to production, and we are now entering one of the more active times of the year. (Our daily page views for a Sunday -- our busiest day -- are over 29 million, and the normal load on our database servers is over a million queries per minute.)

You can find more details on the current state of the Archive's servers below, along with a rough timeline of the issues we experienced between Thursday, May 10, and Monday, May 14. However, the main takeaway is these issues are likely to continue until the Elasticsearch upgrade is completed and our network capacity is increased. We're very grateful for the support and patience you've shown, and we look forward to finishing our upgrades so we can provide you with a stable Archive once more.

Background: Server state

We normally have five Elasticsearch servers, but late last year we turned one of our front end machines into an Elasticsearch server, allowing us to divide these six machines into two groups: one three-machine cluster for the production site, and another for testing the upgraded code.

Having only three Elasticsearch servers meant the site experienced significant issues, so on April 11, we reprovisioned one of our old database servers, which had been producing web pages, as an Elasticsearch server in the production cluster.

In addition to the ongoing Elasticsearch upgrade, our Systems team recently completed a major overhaul intended to help with our long term stability and sustainability. Between November 2017 and March 2018, they reinstalled all the application servers, web front ends, and new Elasticsearch systems with a new version of the Debian (Stretch) operating system using FAI and Ansible. This meant rewriting the configuration from the ground up, since we had previously used FAI and CFEngine. They also upgraded various other packages during this process, and now all that's left to upgrade for the Archive are the database servers.

Timeline

May 10

16:25 UTC: We deploy the code update that will allow us to run the old and new Elasticsearch code simultaneously. (We know the new version still has a few kinks, and we expect to find more, so we're using a Redis-based system called rollout to make sure internal volunteers get the new code while everyone else gets the old version.) Because this is our first deploy since the application servers have been reinstalled, the deploy has to be done by hand.

16:56 UTC: We turn on the new Elasticsearch indexing.

21:03 UTC: We notice -- and fix -- some issues with site skins that resulted from doing a manual deploy.

May 11

05:00 UTC: We see large amounts of traffic on ao3-db06, which is both the Redis server we use for Resque and the MySQL server responsible for writes. We mistakenly believe the traffic is caused by the number of calls to rollout to check if users should see the new filters.

05:36 UTC: We increase the number of Resque workers.

10:06 UTC: The Resque queue is still high, so we increase the number of workers again.

21:00 UTC: We no longer believe the increased traffic is due to rollout, so we turn the new indexing off and schedule 45 minutes of downtime for 06:15 UTC the following morning.

May 12

06:15 UTC: In order to mitigate the extra traffic, we move Redis onto a second network interface on ao3-db01. However, routing means the replies return on the first interface, so it is still overwhelmed.

06:42 UTC: We extend the downtime by 30 minutes so we can change the new interface to a different network, but replies still return on the wrong interface.

07:26 UTC: Since we've used up our downtime window, we roll the change back.

After that, we spend large parts of the day trying to figure out what caused the increase traffic on ao3-db06. With the help of packet dumps and Redis monitoring, we learn that indexing bookmarks on new Elasticsearch is producing a large number of error messages which are stored in Redis and overwhelming the network interface.

May 13

Our coders spend most of Sunday trying to determine the cause of the Elasticsearch errors. We look at logs and try a number of solutions until we conclude that Elasticsearch doesn’t appear to support a particular code shortcut when under load, although it's not clear from the documentation why that would be.

20:45 UTC: We change the code to avoid using this shortcut and confirm that it solves the issue, but we do not resume the indexing process.

23:45 UTC: The Resque Redis instance on ao3-db06 freezes, likely due to load. As a result, some users run into errors when trying to leave comments, post works, or submit other forms.

May 14

06:30 UTC: We restart Redis, resolving the form submission errors. However, we begin to receive reports of two other issues: downloads not working and new works and bookmarks not appearing on tag pages.

16:25 UTC: To help with the download issues, we re-save our admin settings, ensuring the correct settings would be in the cache.

16:34 UTC: Now we look into why works and bookmarks aren't appearing. Investigating the state of the system, we discover a huge InnoDB history length (16 million rather than our more normal 2,000-5,000) on ao3-db06 (our write-related MySQL server). We kill old sleeping connections and the queue returns to normal. The server also returns to normal once the resultant IO has completed.

16:55 UTC: Bookmarks and works are still refusing to appear, so we clear Memcached in case caching is to blame. (It's always -- or at least frequently -- caching!)

17:32 UTC: It is not caching. We conclude Elasticsearch indexing is to blame and start reindexing bookmarks created in the last 21 hours.

17:43 UTC: New bookmarks still aren't being added to tag listings.

17:54 UTC: We notice a large number of Resque workers have died and not been restarted, indicating an issue in this area.

18:03 UTC: We apply the patch that prevents the bookmark indexing errors that previously overwhelmed ao3-db06 and then restart all the unicorns and Resque workers.

18:43 UTC: Once everything is restarted, new bookmarks and old works begin appearing on the tag pages as expected.

19:05 UTC: The site goes down. We investigate and determine the downtime is related to the number of reindexing workers we restarted. Because we believed we had hotfixed the issue with the reindexing code, we started more reindexing workers than usual to help with the indexing process. However, when we started reindexing, we went above 80% of our 1 Gbit/sec of ethernet to our two MySQL read systems (ao3-db01 and ao3-db05).

19:58 UTC: After rebalancing the traffic over the two read MySQL instances and clearing the queues on the front end, the indexers have stopped, the long queues for pages have dissipated, and the site is back.

Takeaways

  • We will either need multiple bonded ethernet or 10 Gbit/sec ethernet in the very near future. While we were already expecting to purchase 10 Gbit networking in September, this purchase may need to happen sooner.
  • Although it has not been budgeted for, we should consider moving Redis on to a separate new dedicated server.

While we are running with reduced capacity in our Elasticsearch cluster and near the capacity of our networking, the reliability of the Archive will be adversely affected.

Are you Canadian? OTW Legal Wants Your Stories!

http://archiveofourown.org/admin_posts/10512

Spotlight on Legal Issues

Are you Canadian? OTW Legal wants your stories about being a fan!

Over the years, OTW Legal has spoken for fans and fanwork creators in comments to governments around the world including the U.S., the E.U., Canada, Australia, and South Africa. And we want your help to keep doing that! The Canadian government is currently conducting a review of the Canadian Copyright Act. The Parliamentary Committee responsible for the review has already received some comments complaining about fan-friendly laws like the 2012 expansions to fair dealing and the User-Generated Content exception to copyright infringement. OTW Legal wants to show Parliament the other side of the story: the important value that Canadian fanwork creators get from being able to create transformative works.

Are you Canadian and have you expressed yourself, gained skills, been part of creative communities, or otherwise experienced the benefits of being able to create transformative works--works that are legal to create in Canada because of fair dealing and the UGC exception? If so, OTW Legal would love to hear your stories. We need to submit our comments soon, so please send our Legal Advocacy team your stories about how being able to create fanworks and belong to fan communities has helped you, by the end of May. (Feel free to use a pseudonym if you don't want us to share your personally identifying information.) We’ll use your stories to support our legal advocacy work in Canada and worldwide.

Thanks!

Five Things Claire Baker Said

http://archiveofourown.org/admin_posts/10491

Five Things an OTW Volunteer Said

Every month or so the OTW will be doing a Q&A with one of its volunteers about their experiences in the organization. The posts express each volunteer's personal views and do not necessarily reflect the views of the OTW or constitute OTW policy. Today's post is with Claire Baker, who volunteers as a member of the OTW Board.

How does what you do as a volunteer fit into what the OTW does?

The OTW's current Board members tend to wear many hats, and I am no exception. I co-chair the AO3 Documentation Committee (call us Docs!), wrangle tags in a handful of fandoms, do layout editing for Transformative Works and Cultures (TWC), and also serve on our Board of Directors.

Docs and Tag Wrangling are both AO3 committees: Docs writes FAQs, tutorials, and other user-facing help documentation, and Tag Wrangling sorts the tags users put on their works so that all the works about the same topic (fandom, character, pairing, etc) can be easily found. With TWC, I'm on the production team: my job is to take articles that have been written and copyedited, and add html formatting.

The Board of Directors oversees everything, though this oversight works best when we have direct communication with those who will be affected by our work. As such, we end up meeting either synchronously or asynchronously with chairs and committee members on a regular basis. We're aiming to build strong connections between us and each part of the OTW. And if we're not, I hope the committees can lead us to understand how we better can do our job!

What is a typical week like for you as a volunteer?

Like many people say in these Five Things posts, there's no such thing as a typical week within the OTW. However, I do have a schedule that I like to follow when I can.

I often try to start my OTW day with some tag wrangling. This may involve sorting through new tags, spreadsheeting older ones, or double-checking that canonical tags for my fandoms are all in order.

After that, I'll do some work for Docs. As a chair, this usually means looking at AO3 FAQs or tutorials that have recently been drafted, or documents that have completed our editing cycle and are almost ready for upload. Ideally, I'd look at one or more of these a day, but generally it's a little less than that. We chairs rotate between regular administrative tasks, so I may also send out our weekly check-in or a reminder to our sister committees that works are open for external betaing. Chairs attempt to meet weekly and talk asynchronously around that, so in a typical week I've been in contact with both of my co-chairs, and we've likely discussed or done work on a bigger administrative task as well.

If TWC has a new issue coming up, I'll spend a couple of days applying html to articles in order to get them publication ready, but that's not necessarily in a typical week; my job is only needed a couple of times per year.

And then there's my Board work. The most regular part here is voting. As soon as a request comes in, we will look into the issue, discuss it as needed, and vote. Voting happens asynchronously, and we are usually able to respond within 48 hours. Meetings are very much of a Board reality, and so chances are that I've scheduled or am scheduling one during any given week. (And on the weeks when Board does not meet, I will likely have a meeting scheduled with one of my other committees.) Outside of that, I try to put in an hour on Board documentation somewhere in my day —- either looking over existing pages, or drafting missing documentation. I've still got a lot to learn, but I'll keep on studying, asking questions, and working with others to build a deeper understanding for myself, and hopefully a better OTW for everyone.

Board members hold the only elected positions in the OTW. What made you decide to run?

The short answer is that I'm a nerd who loves the OTW and wants to see it become the best it can be. The long answer's a little more complicated, if no less heartfelt.

I fell in love with the concept of the OTW the moment that I was introduced to it in 2012. I fell in love with the people when I started volunteering in 2014. I was happy to do various tasks and to lend an opinion when needed, and I built a reputation of being a good person with strong leadership skills somewhere along the line. Meanwhile, I was witnessing transition: Docs went from a workgroup to a committee, the OTW adopted a new internal communication platform, and the Board itself went through a changing of the guard. Through all of that, I learned how great (and complicated) the OTW could be, and how much I valued it as a place of work.

By 2017, I was the third-longest serving member in Docs, and had gained a breadth of experience through mentoring new staff, taking on new roles within the OTW, and generally being an active participant in our volunteer community. When Elections and the Board started running opportunities to learn more about candidacy, I found myself participating there too.

In all honesty, I originally expected to wait one more year to have that much more knowledge under my belt, and so I could run alongside a friend who didn't quite qualify for candidacy. However, there was a need for more candidates for a fair, contested election, and I knew I had the time and skill set needed to serve the OTW well, so I put my name up for consideration.

Now, I wouldn't take back that decision for anything. I work alongside people I think of as great role models, and learn more from them every time we talk. I've gained a lot of knowledge about the OTW as a whole as well, and really do enjoy both the joys and challenges that come with helping the entire organization move forward smoothly. My hope now is that I can help foster the next generation of people to join our ranks, whether as new volunteers or new board members, and make the OTW an even better place for the fan community at large.

What are things you think fans probably do and don't understand about the OTW Board?

When I was new to the OTW, there was a lot of wariness about Board, and a pretty strong Them vs. Us mentality. We're working on breaking this down, but it takes a lot of time and effort to build, earn, and maintain trust.

The Board exists to make sure that we're all on the same page, and that we're doing what we need to still be around for years to come. If there's something coming up that we need to be prepared for, like GDPR, we'll make sure that the necessary conversations are happening. If AO3 needs more servers, we're here to make sure that those needs are acknowledged and met. Otherwise, we're happy to talk about our favourite characters and ships and take part in a wide variety of fannish activities, just like everyone else here.

What fannish things do you like to do?

I'm a cosplayer. I entered competitions in my first few years of cosplay, and won an award or two, but have since found that I prefer doing it a little more casually. I also run panels at conventions, and often end up coordinating panel and cosplay schedules alike for the group that I'll be attending with. Really, there's nothing better than spending a weekend with your friends, talking about your favourite fandoms while you're all dressed as characters from them. It's a lot of fun, and well worth the effort.

I also write fanfic and RP, and love sharing headcanons with my friends. Somehow this led to becoming a regular beta for a few of them, which I love. Being able to look at their works before anyone else is an absolute treat, and if I can help make their works stronger before they're published, all the better.

Most of my writing these days is for gift exchanges. I have a habit of running three or four small gift exchanges simultaneously, and participating in several more. I'm also a serial pinch hitter, and will do my best to make sure everyone has something to look forward to when gifts are revealed.

My other love is for the academic side of fandom. I'm building up a small library of books related to the subject, and would love to get a Masters or PhD in fan studies one day. In the meantime, I'm enjoying my time as an independent scholar.


Now that our volunteer’s said five things about what they do, it’s your turn to ask one more thing! Feel free to ask about their work in comments. Or if you'd like, you can check out earlier Five Things posts.

The Organization for Transformative Works is the non-profit parent organization of multiple projects including Archive of Our Own, Fanlore, Open Doors, Transformative Works and Cultures, and OTW Legal Advocacy. We are a fan run, entirely donor-supported organization staffed by volunteers. Find out more about us on our website.