Ubuntu
For Ubuntu, a *lot* of time and effort goes into triaging X bugs. Mostly, this work consists of helping bug reporters get enough data to actually begin troubleshooting the problem. Sadly, a lot of bugs go uninvestigated due to lack of enough info.
I have noticed lately though, that a number of reporters are starting to include the info needed, which is great! Developers can then focus their time in actually solving their problem rather than requesting files and trying to think about what to ask for that might give clues.
If nothing else, the #1 most important thing to always include when reporting an X bug is your /var/log/Xorg.0.log. This file is chock full of useful data... versions of modules, configs, resolutions identified, warnings, error messages, and sometimes even partial backtraces. If you remember nothing else, please remember this one thing: "When reporting an X bug, ALWAYS include your Xorg.0.log".
Sometimes Xorg.0.log isn't necessary, but it seems like 80-90% of the time I need some bit of data which is there, so unless you're *certain* it's not needed, make every attempt to include it. If you can include it directly with your report at the time you submit it, you can save triagers time and decrease the time before someone takes a stab at troubleshooting your problem.
The #2 most important thing to include depends a bit on what type of X bug you've encountered. I'd group X bugs like this:
Screen Corruption. A photograph of the screen helps a lot, since it's often hard to describe in words, but can be obvious when you see it.
X Crashes/Lockups/Freezes. Many people report these kinds of X breakages; certainly they are troubling, and unfortunately not as rare as we'd like to make them. The important thing to include is a full backtrace with debug symbols installed (see the directions). Indeed, there really is *nothing* anyone can do to troubleshoot this class of bug without a backtrace, and we'll usually close these bug reports if no backtrace is posted in a reasonable amount of time.
If you're not familiar with what a backtrace is, it's basically a list of the function calls that were made immediately leading up to the crash. A 'full' backtrace also includes the values of the function's parameters and local variables, which is useful for spotting NULL pointers, invalid strings or indexes, etc.
I can't emphasize the importance of full backtraces enough. I've noticed that somewhere around 20-30% of the time where a full backtrace is posted, we're able to identify the bug and propose a patch directly. Often the bugs are simple null pointer dereferences or other obvious coding issues, we can patch immediately and post upstream. Without a backtrace there would be no way to identify these fixes.
"But it worked in Gutsy...?". X changes a lot. New functionality is continually being added and we want to make the new stuff available as we put out new Ubuntu's. Unfortunately, sometimes new code means new bugs. I see a lot of irritation when people find that something that worked fine in the previous stable release suddenly stops working in the development release.
These kind of bugs have a surprisingly effective brute force way to isolate the problem: Just start checking some of the intermediary versions between the stable and development releases. (Usually this involves building from upstream's git repository, perhaps using 'git bisect'. Stay tuned, I have ideas on how to make this easier for folks.)
Everything Else. Of course, there are many other kinds of failures... configuration errors, wrong DPI's, performance problems, and on and on. Use your judgment in what to include in addition to Xorg.0.log. Sometimes experimenting with alternate xorg.conf Options is worth doing. Sometimes checking with a different monitor, or with Compiz turned on or off can produce useful info. More tips are on the Reporting X Bugs page.
Posted in Ubuntu Submitted by bryce on Mon, 2008-04-21 22:30.
bryce's blog
Thought I'd take a break from the Xrandr GUI blogging to make mention of a few new (and old) tools I put together for the Xorg bug swat team.
First up is a new high level Ubuntu-X Status Page, updated several times an hour. This was directly inspired by Leann Ogasawara's Ubuntu Development Weather Report tool, but is focused one level down on just Xorg stuff.
This has been a handy tool for reviewing and accessing various "targets of opportunity" lists that others maintain, like Daniel's sponsoring and 'really-fix-it' pages, and Brian's handy 'bugs with more than 5 subscribers' pages, along with the usual milestone and release targeted bug lists in launchpad.
One thing this highlights is that our -ati driver needs a lot more bug triager attention, as there are quite a few serious bugs plaguing users (note the large number of subscribers).
Anyway, the tool is still a bit crude, but it's been useful to me. I'm hoping to develop it a bit further in Intrepid.
Next up is an older tool I've mentioned, the Launchpad Bug Graphs. I don't know if anyone else uses it, but I use it daily.
The 6-month view of the -intel X driver Open Bugs shows the great progress we've made with the Intel bug stats. I'd set as a goal for Hardy to reduce it to below 100, and after quite a bit of work, this was achieved! Of course, 100 bugs in -intel is still concerning, and I hope with Ubuntu community member's help we can continue pushing this down.
I think -ati could benefit from a similar intense focus; while the total raw number of bugs has not grown significantly since last release, the numbers are higher than they've been for as long as we've been graphing them, which is troublesome given that we're about to finalize an LTS. We could really use some serious help here, so if you're an ATI user looking for a way to really help Ubuntu, this could be it.
The next tool to show off is extremely new, and in fact is really just a static prototype, but has already been proven useful. I'm calling it the Historical Ubuntu Xorg Intel Drivers page, but that doesn't really roll off the tongue. Anyway, it's essentially a series of git snapshots of Xorg's upstream Intel driver, pre-built for hardy-i386.
There is a certain class of bugs I've noticed, where the user reports that the issue worked in Gutsy or an earlier Alpha, but currently is broken, and this tool is designed to assist in narrowing this down. The user identifies a "known good" and a "known bad" entry in the table and then "bisects", picking a deb halfway between them, and tests that. Then, depending on whether that failed or not, they pick one halfway between it and one of the original two, and so on until they identify the two adjacent versions between which it fails.
I didn't build every single git commit, since that would be a bit overwhelming. I skipped every several commits, figuring that once it's down to about 4-8 patches then a developer ought to be able to pick out the most likely candidate. I hope this will help enable non-technical Ubuntu folk to do much of the legwork in isolating regressions, without needing to break out a compiler or spend a lot of time on it. (As a side benefit, this gives them a way to upgrade or downgrade to a working driver if worse comes to worse.)
When I get more time I plan on enhancing this tool with debs for more releases, for amd64, and for the -ati and -nv drivers, and maybe -radeon if there's sufficient interest. We'll see how it works.
Okay, last tool. Well, documentation actually... This is the Ubuntu X page on the Ubuntu wiki. I've linked in tons of other useful stuff here, and in particular would like to highlight the X Debugging Handbook, which I've recently broken up into separate pages for easier reading. Through the Hardy bug fixing period I've added a good bit to the Troubleshooting page, to document steps one can take for figuring out various classes of problems. It's not quite to the paint-by-numbers level that I hope to get it to, but getting close.
Posted in Ubuntu Submitted by bryce on Fri, 2008-04-11 00:37.
bryce's blog
Like kees mentioned, Kees and I tested his laptops on the Hitachi projector that Elmo lent me from work.
The background here is pretty simple: We want Linux to work with projectors, or at *least* for Ubuntu to work with Canonical's specific projectors. It's embarrassing enough as a presenter to walk into a Linux conference providing some random projector and not be able to connect; imagine how frustrating it is to go to a Ubuntu conference with an up to date Ubuntu laptop, and plug into Canonical's own projector and not be able to get it to work!
Historically, getting a laptop to work with a projector has been a Dark Art, requiring major xorg.conf tweakage. Things have gotten better with the introduction of Xrandr, though.
Intel graphics was the first to get good Xrandr support. Both my (working) laptops are Intel based, and sadly I could find no bugs with the projector. Hotplugging, resolution switching, rotation... it all just worked.
Kees had described some rather nasty issues he'd found with his ATI graphics laptop, so I knew there were resolution issues, I just had to find them!
So a couple weeks ago I ordered several popular (but cheap) nVidia 7000-series and ATI cards, and tested 9 different hardware configurations.
I did find some bugs - rotation on -ati is **horribly** busted, for example - but was surprised to see that by and large, resolution changing worked amazingly well. Hotplugging not so much. But I never had to fiddle with xorg.conf. Most of the issues I did find were oddball corner cases, or were easily fixed by doing a vt switch, or switching to another resolution and back, or etc.
So today I had Kees bring his laptops over, including the aforementioned broken ATI one, which he also upgraded to Hardy today. The other projector is a Dell with a recent 8000-series nVidia graphics chipset.
Amazingly, he was able to boot the projector on both laptops to just about every resolution supported. There were a handful of various little glitches and issues, but mostly known bug - I mentioned to him that I felt like he was demoing everything currently in Launchpad for Xorg. Even rotation sort of worked for him on -ati: We could invert the screen without problems, but Left/Right didn't work and required restarting X.
Of course, we did nearly all of this testing with the new Xrandr GUI, which aside from a couple issues fulfilled its role admirably. I was particularly gratified seeing the Revert Dialog (which I'd only just uploaded last night) came up and worked exactly as expected.
Unfortunately, while we were pleased with X and the Xrandr GUI at letting us experiment with all these resolutions on the fly, it's clear that things beyond X need work. GNOME's panels were quite easy to get confused. The desktop background can get strangely stretched in some configs. Full screen apps (games and movie players) also seem to get easily confused; they tend to want to display on the laptop, when of course you want them to go up on the projector!
I guess my hope is that providing an easier way for people to experiment with their Xrandr settings, it will better expose these WM-level and app-level issues and make it simpler to test them, and maybe we'll start seeing improvements in Intrepid and later. I'll look forward to seeing games and movies display properly on the expected output.
Posted in Ubuntu Submitted by bryce on Sun, 2008-04-06 08:24.
bryce's blog
I'm sure everyone must be sick of hearing about this xrandr gui work, but by popular demand I've added a Revert Dialog, like in the old Screen Resolution tool. Needs a bit more testing.
Posted in Ubuntu Submitted by bryce on Wed, 2008-04-02 04:57.
bryce's blog
If you asked me what the worst day in the world is to announce an open source project's release, April 1st would be it. Yet here we are, and the announcements for 0.46 are hitting the wire now.
Actually, our true release was about a month ago, but we had to delay announcements because of a series of bad problems with the Windows package. I set an ultimatum that we would be announcing on March 24th regardless - the ultimatum worked at getting fixes in for all the issues, but we still didn't have them integrated into a working .EXE. But thanks to a bunch of new and old Windows contributors, in a week all this got put together and sorted out... just in time for April Fools Day. Heh.
Anyway, make up a Windows rant and insert it here. An apropos one for today would have something to do with open source projects not supporting monopolies that subvert ISO standards processes. But pick your favorite, I'm sure I'd agree.
Meanwhile, go get your Inkscape 0.46 from our website. There's been a year's worth of awesome features and fixes added to it since our 0.45.1 release.
I'm thrilled to see mailing list support added to Launchpad! We switched Inkscape over to Launchpad for bug tracking
a while back and the community has been loving it.
Our mailing lists are currently hosted at SourceForge. Most of the time they work okay, but there's a lot of little annoyances. Archives are hard to browse and usually not up to date. Administration is a hassle, generally requiring a dozen clicks just to check someone's subscription status. And so on. So I'm really interested in test driving Launchpad's lists and see how they stack up. From what I've seen so far it looks damn sweet. The integration with teams is brilliant and gives *so* much better organization potential than what can be done with our current tools.
So, thinking further afield, what other Launchpad features would nicely solve Inkscape's infrastructure needs ?
First and foremost on the list would be file hosting. When we put out a release we get tons and tons of downloads, especially of the windows packages. Putting an Inkscape .exe on a "normal" server is basically giving it a death sentence; we are absolutely dependent on mirroring. PPAs really aren't suitable, and really are intended to solve a different, specific problem. What we need is just plain old mirror distribution. It would be nice to have access control to limit who can "officially" post packages for a given platform, so like only our OSX packagers can post OSX packages, and so on.
Somewhat along with this, we do thrice-daily automated builds of our SVN tree, and build packages for various platforms. Indeed, a lot of our hard core users eschew our official releases in favor of the dev builds. This works great for them - they risk bugs but gain access to the latest features no one else has. And for us it works great because we get swift feedback when things break. We don't keep every built version around, but it's important to keep some older versions available in case the latest version is completely trashed.
So it would be wonderful to have an automated build functionality in Launchpad that we could hook into for doing all of our builds (not just Ubuntu, but also RPMs, EXEs, and DMGs). One extra nicety would be an ability for people to tag particular builds, to flag ones that have issues, so other downloaders can avoid them.
Next on the wishlist would have to be project forums. While I am a dyed in the wool mailing list user, we have *tons* of users that constantly ask about forums.
Web hosting is another area high on the needs list. In particular, we'd like to see drupal hosting, and/or mediawiki.
Statistics are another feature we'd love to see, particularly download numbers/charts. I used to script a lot around SF to extract interesting metrics and stats, but frequent UI changes kept breaking my screen scrapers.
Posted in Inkscape | Ubuntu Submitted by bryce on Fri, 2008-03-28 19:35.
bryce's blog
Here's the latest version of the Xrandr GUI, that'll be uploaded post-beta.
We've taken care of the major show stopper issues by this point. There's still a few driver-specific xrandr bugs (like lockups when rotating) yet to be sorted out in the coming weeks and months.
You can access this tool via the 'Screen Resolution' link in preferences, or on the command line via 'gnome-display-properties'.
There are obviously still a lot of features to be added, such as being able to lay out multi-head setups. These are in the works, although I'm not certain if they'll be provided by default in Hardy. But I'll be planning on providing updates via a PPA, and possibly we can investigate including it for 8.04.1 if it looks stable and if there is strong interest.
Meanwhile, I've also disabled displayconfig-gtk in the menus. It is still available from the command line (we still need it for bulletproof-X mode). It's become largely obsolete due to Xorg 7.3 autoconfiguration changes, which has resulted in a lot of stuff that used to be set in xorg.conf to no longer be listed there. This confuses displayconfig-gtk greatly, and has been the source of a lot of bug reports and trouble. Further, while displayconfig-gtk has a nice multi-head layout capability, it depends on (the now heavily deprecated) Xinerama, yet these days there are very few drivers that still use Xinerama, so it's become obsolete on this count as well (indeed, you can produce broken xorg.conf's this way). So, the number of people who can still make use of it is becoming quite small, but is not yet negligible, so hopefully having this tool available on the system will still address these users' needs.
Posted in Ubuntu Submitted by bryce on Sat, 2008-03-22 00:03.
bryce's blog
One of the first bugs that showed up after deploying the Xrandr GUI was pretty serious - gnome-settings-daemon crashed on certain X drivers (radeonhd and intel with i855 chips most notably, also fglrx and possibly nvidia). This was an unfortunate bug in that it prevented login. Fortunately, it was straightforward to work around - just disable the xrandr plugin for gnome-settings-daemon via gconf (/apps/gnome_settings_daemon/keybindings/xrandr)
Unfortunately this error has proven difficult to debug. The issue is that while several drivers support xrandr 1.2, they still crash when this tool tries to set up xrandr. So simply checking for xrandr versions via XRRQueryVersion() is insufficient.
I gather that fglrx has recently gained xrandr 1.2 support (at least, in tests using the xrandr command line tool, it accepts xrandr 1.2 syntax for changing screen size, but we got a BadMatch X error when doing so).
In the case of -intel i855, it's interesting that there have been no reports of issues on newer chipsets (in fact, it works fine on my 945 and 065 systems). The i855 and earlier chips have some unique bios issues that had been limiting people to use of -i810 until recently; it's entirely conceivable that there could be some remaining bugs particular to xrandr 1.2 support.
radeonhd is the most perplexing, since it advertises good xrandr 1.2 support. I found that there were a number of patches submitted to xserver by one of the radeonhd developers to fix various xrandr problems, but testing with those patches included did not change behavior at all.
I'm not sure how to troubleshoot this issue further. The most obvious next step is brute force - detect if we're running with one of these drivers or chipsets, and just avoid doing the xrandr calls in this case.
Unfortunately, xorg does not provide an interface for getting the driver name, but it can be parsed from the Xorg.0.log. I wrote a quick code to do this, which might be useful for other purposes.
I wonder if this issue might be related somehow to this bug?
Posted in Ubuntu Submitted by bryce on Tue, 2008-03-11 23:56.
bryce's blog
Finally today we've uploaded the new Xrandr GUI to Ubuntu Hardy. I mentioned working on a tool a couple weeks ago; I discovered that Soren Sandmann was also working on such a tool and was further along, so I shifted focus to getting that tool ready for integration in Ubuntu.
The above photo shows two laptops, one connected to an external monitor, the other to a projector; resolutions, rotations, and so on can all be handled on a per-monitor basis. The projector image is hard to see since it's so bright, but it's working fine with these laptops.
This tool also has a Cairo-based graphical layout to show the screens; notice how it detects and displays the monitor's name.
You'll notice the 'clone screens' checkbox; currently the utility does not have a way to specify left-of/right-of, so it just does cloning for now. It's entirely within the design scope of this tool to do that, so expect to see that capability in coming months. (For now, I'm going to leave the clone checkbox out).
The tool allows rotation as well.
One issue I found is that it only allows making changes to one screen at a time. If you need to change the other monitor's settings, I find it works to disable the first one (set resolution to 'off'), make the changes to the second, and then re-enable the first.
CAUTION: Xrandr is buggy on certain hardware. It's not this GUI tool's fault (you can usually replicate the bugs using the 'xrandr' command-line tool), but the GUI increases the exposure to these bugs since it's so easy to play with them now. For instance, some hardware locks up or performs badly when rotated. Others exhibit lockups or corruption when scaling up or scaling down. So, if you want to experiment, please be sure to save your work first. ;-) Bugs should be reported against your video driver, not against xrandr or this gui.
Posted in Ubuntu Submitted by bryce on Fri, 2008-02-29 23:15.
bryce's blog
Periodically, someone on the inkscape list proposes the idea of a simplified inkscape interface, to make it more child friendly or whatnot.
When I was fiddling with ume and playing with the EEE PC I wondered about a simplified inkscape for subnotebook and embedded hardware, and what it'd take to achieve.
So, in a couple hours of ifdeffing out various things, this is what I came up with:
The main challenge was the aux toolbar - I think we'll need JonCruz's toolbar rework (which missed 0.46 but will definitely be in for 0.47); I just hacked it down. Deciding what to keep in the left toolbox was a tough choice, but I think I kept the most useful tools; the paint bucket would be the next on the chopping block, but I understand kids love it, so it just made it.
You'll note I turned off the scrollbars, ruler, and color palette - this was trivial to do via the menus, and could be turned back on as desired.
I tried getting rid of the grippy handles for the toolboxes, since for this use case people probably wouldn't need to be tearing off toolbars, and it'd save a touch of space both horizontally and vertically.
One thing I noticed as an issue is that some of the tool panels on the right take up a large proportion of the canvas space when at 640x480. A few of the dialogs are rather big too.
So, I think we could quite easily support an 'inklite' version of inkscape, built from the same codebase. Implementation-wise, it'd be easiest to do it as just two different executables via a #define INKLITE type thing; a lot of the menu code and stuff uses structs and static strings that can't be modified at runtime. This is analogous to what we already do for inkview.
Posted in Inkscape | Ubuntu Submitted by bryce on Sat, 2008-02-23 04:06.
bryce's blog
